diff --git a/.github/ISSUE_TEMPLATE/10_question.md b/.github/ISSUE_TEMPLATE/10_question.md index 6e23fbdc605..a112b9599d5 100644 --- a/.github/ISSUE_TEMPLATE/10_question.md +++ b/.github/ISSUE_TEMPLATE/10_question.md @@ -7,6 +7,6 @@ assignees: '' --- -Make sure to check documentation https://clickhouse.yandex/docs/en/ first. If the question is concise and probably has a short answer, asking it in Telegram chat https://telegram.me/clickhouse_en is probably the fastest way to find the answer. For more complicated questions, consider asking them on StackOverflow with "clickhouse" tag https://stackoverflow.com/questions/tagged/clickhouse +> Make sure to check documentation https://clickhouse.yandex/docs/en/ first. If the question is concise and probably has a short answer, asking it in Telegram chat https://telegram.me/clickhouse_en is probably the fastest way to find the answer. For more complicated questions, consider asking them on StackOverflow with "clickhouse" tag https://stackoverflow.com/questions/tagged/clickhouse -If you still prefer GitHub issues, remove all this text and ask your question here. +> If you still prefer GitHub issues, remove all this text and ask your question here. diff --git a/.github/ISSUE_TEMPLATE/20_feature-request.md b/.github/ISSUE_TEMPLATE/20_feature-request.md index 99a762a2019..f59dbc2c40f 100644 --- a/.github/ISSUE_TEMPLATE/20_feature-request.md +++ b/.github/ISSUE_TEMPLATE/20_feature-request.md @@ -7,16 +7,20 @@ assignees: '' --- -(you don't have to strictly follow this form) +> (you don't have to strictly follow this form) **Use case** -A clear and concise description of what is the intended usage scenario is. + +> A clear and concise description of what is the intended usage scenario is. **Describe the solution you'd like** -A clear and concise description of what you want to happen. + +> A clear and concise description of what you want to happen. **Describe alternatives you've considered** -A clear and concise description of any alternative solutions or features you've considered. + +> A clear and concise description of any alternative solutions or features you've considered. **Additional context** -Add any other context or screenshots about the feature request here. + +> Add any other context or screenshots about the feature request here. diff --git a/.github/ISSUE_TEMPLATE/40_bug-report.md b/.github/ISSUE_TEMPLATE/40_bug-report.md index 5c8611d47e6..d62ec578f8d 100644 --- a/.github/ISSUE_TEMPLATE/40_bug-report.md +++ b/.github/ISSUE_TEMPLATE/40_bug-report.md @@ -7,11 +7,11 @@ assignees: '' --- -You have to provide the following information whenever possible. +> You have to provide the following information whenever possible. **Describe the bug** -A clear and concise description of what works not as it is supposed to. +> A clear and concise description of what works not as it is supposed to. **Does it reproduce on recent release?** @@ -19,7 +19,7 @@ A clear and concise description of what works not as it is supposed to. **Enable crash reporting** -If possible, change "enabled" to true in "send_crash_reports" section in `config.xml`: +> If possible, change "enabled" to true in "send_crash_reports" section in `config.xml`: ``` @@ -39,12 +39,12 @@ If possible, change "enabled" to true in "send_crash_reports" section in `config **Expected behavior** -A clear and concise description of what you expected to happen. +> A clear and concise description of what you expected to happen. **Error message and/or stacktrace** -If applicable, add screenshots to help explain your problem. +> If applicable, add screenshots to help explain your problem. **Additional context** -Add any other context about the problem here. +> Add any other context about the problem here. diff --git a/.github/ISSUE_TEMPLATE/50_build-issue.md b/.github/ISSUE_TEMPLATE/50_build-issue.md index 73c97cc3cfb..a358575cd7c 100644 --- a/.github/ISSUE_TEMPLATE/50_build-issue.md +++ b/.github/ISSUE_TEMPLATE/50_build-issue.md @@ -7,10 +7,11 @@ assignees: '' --- -Make sure that `git diff` result is empty and you've just pulled fresh master. Try cleaning up cmake cache. Just in case, official build instructions are published here: https://clickhouse.yandex/docs/en/development/build/ +> Make sure that `git diff` result is empty and you've just pulled fresh master. Try cleaning up cmake cache. Just in case, official build instructions are published here: https://clickhouse.yandex/docs/en/development/build/ **Operating system** -OS kind or distribution, specific version/release, non-standard kernel if any. If you are trying to build inside virtual machine, please mention it too. + +> OS kind or distribution, specific version/release, non-standard kernel if any. If you are trying to build inside virtual machine, please mention it too. **Cmake version** diff --git a/contrib/AMQP-CPP b/contrib/AMQP-CPP index 03781aaff0f..1a6c51f4ac5 160000 --- a/contrib/AMQP-CPP +++ b/contrib/AMQP-CPP @@ -1 +1 @@ -Subproject commit 03781aaff0f10ef41f902b8cf865fe0067180c10 +Subproject commit 1a6c51f4ac51ac56610fa95081bd2f349911375a diff --git a/contrib/amqpcpp-cmake/CMakeLists.txt b/contrib/amqpcpp-cmake/CMakeLists.txt index 4e8342af125..5637db4cf41 100644 --- a/contrib/amqpcpp-cmake/CMakeLists.txt +++ b/contrib/amqpcpp-cmake/CMakeLists.txt @@ -10,11 +10,12 @@ set (SRCS "${LIBRARY_DIR}/src/deferredconsumer.cpp" "${LIBRARY_DIR}/src/deferredextreceiver.cpp" "${LIBRARY_DIR}/src/deferredget.cpp" - "${LIBRARY_DIR}/src/deferredpublisher.cpp" + "${LIBRARY_DIR}/src/deferredrecall.cpp" "${LIBRARY_DIR}/src/deferredreceiver.cpp" "${LIBRARY_DIR}/src/field.cpp" "${LIBRARY_DIR}/src/flags.cpp" "${LIBRARY_DIR}/src/linux_tcp/openssl.cpp" + "${LIBRARY_DIR}/src/linux_tcp/sslerrorprinter.cpp" "${LIBRARY_DIR}/src/linux_tcp/tcpconnection.cpp" "${LIBRARY_DIR}/src/inbuffer.cpp" "${LIBRARY_DIR}/src/receivedframe.cpp" diff --git a/contrib/arrow b/contrib/arrow index debf751a129..078e21bad34 160000 --- a/contrib/arrow +++ b/contrib/arrow @@ -1 +1 @@ -Subproject commit debf751a129bdda9ff4d1e895e08957ff77000a1 +Subproject commit 078e21bad344747b7656ef2d7a4f7410a0a303eb diff --git a/contrib/arrow-cmake/CMakeLists.txt b/contrib/arrow-cmake/CMakeLists.txt index 2237be9913a..2c72055a3e7 100644 --- a/contrib/arrow-cmake/CMakeLists.txt +++ b/contrib/arrow-cmake/CMakeLists.txt @@ -194,9 +194,18 @@ set(ARROW_SRCS "${LIBRARY_DIR}/compute/cast.cc" "${LIBRARY_DIR}/compute/exec.cc" "${LIBRARY_DIR}/compute/function.cc" + "${LIBRARY_DIR}/compute/function_internal.cc" "${LIBRARY_DIR}/compute/kernel.cc" "${LIBRARY_DIR}/compute/registry.cc" + "${LIBRARY_DIR}/compute/exec/exec_plan.cc" + "${LIBRARY_DIR}/compute/exec/expression.cc" + "${LIBRARY_DIR}/compute/exec/key_compare.cc" + "${LIBRARY_DIR}/compute/exec/key_encode.cc" + "${LIBRARY_DIR}/compute/exec/key_hash.cc" + "${LIBRARY_DIR}/compute/exec/key_map.cc" + "${LIBRARY_DIR}/compute/exec/util.cc" + "${LIBRARY_DIR}/compute/kernels/aggregate_basic.cc" "${LIBRARY_DIR}/compute/kernels/aggregate_mode.cc" "${LIBRARY_DIR}/compute/kernels/aggregate_quantile.cc" @@ -207,6 +216,7 @@ set(ARROW_SRCS "${LIBRARY_DIR}/compute/kernels/scalar_arithmetic.cc" "${LIBRARY_DIR}/compute/kernels/scalar_boolean.cc" "${LIBRARY_DIR}/compute/kernels/scalar_cast_boolean.cc" + "${LIBRARY_DIR}/compute/kernels/scalar_cast_dictionary.cc" "${LIBRARY_DIR}/compute/kernels/scalar_cast_internal.cc" "${LIBRARY_DIR}/compute/kernels/scalar_cast_nested.cc" "${LIBRARY_DIR}/compute/kernels/scalar_cast_numeric.cc" @@ -214,15 +224,18 @@ set(ARROW_SRCS "${LIBRARY_DIR}/compute/kernels/scalar_cast_temporal.cc" "${LIBRARY_DIR}/compute/kernels/scalar_compare.cc" "${LIBRARY_DIR}/compute/kernels/scalar_fill_null.cc" + "${LIBRARY_DIR}/compute/kernels/scalar_if_else.cc" "${LIBRARY_DIR}/compute/kernels/scalar_nested.cc" "${LIBRARY_DIR}/compute/kernels/scalar_set_lookup.cc" "${LIBRARY_DIR}/compute/kernels/scalar_string.cc" + "${LIBRARY_DIR}/compute/kernels/scalar_temporal.cc" "${LIBRARY_DIR}/compute/kernels/scalar_validity.cc" + "${LIBRARY_DIR}/compute/kernels/util_internal.cc" "${LIBRARY_DIR}/compute/kernels/vector_hash.cc" "${LIBRARY_DIR}/compute/kernels/vector_nested.cc" + "${LIBRARY_DIR}/compute/kernels/vector_replace.cc" "${LIBRARY_DIR}/compute/kernels/vector_selection.cc" "${LIBRARY_DIR}/compute/kernels/vector_sort.cc" - "${LIBRARY_DIR}/compute/kernels/util_internal.cc" "${LIBRARY_DIR}/csv/chunker.cc" "${LIBRARY_DIR}/csv/column_builder.cc" @@ -231,6 +244,7 @@ set(ARROW_SRCS "${LIBRARY_DIR}/csv/options.cc" "${LIBRARY_DIR}/csv/parser.cc" "${LIBRARY_DIR}/csv/reader.cc" + "${LIBRARY_DIR}/csv/writer.cc" "${LIBRARY_DIR}/ipc/dictionary.cc" "${LIBRARY_DIR}/ipc/feather.cc" @@ -247,6 +261,7 @@ set(ARROW_SRCS "${LIBRARY_DIR}/io/interfaces.cc" "${LIBRARY_DIR}/io/memory.cc" "${LIBRARY_DIR}/io/slow.cc" + "${LIBRARY_DIR}/io/stdio.cc" "${LIBRARY_DIR}/io/transform.cc" "${LIBRARY_DIR}/tensor/coo_converter.cc" @@ -257,9 +272,9 @@ set(ARROW_SRCS "${LIBRARY_DIR}/util/bit_block_counter.cc" "${LIBRARY_DIR}/util/bit_run_reader.cc" "${LIBRARY_DIR}/util/bit_util.cc" - "${LIBRARY_DIR}/util/bitmap.cc" "${LIBRARY_DIR}/util/bitmap_builders.cc" "${LIBRARY_DIR}/util/bitmap_ops.cc" + "${LIBRARY_DIR}/util/bitmap.cc" "${LIBRARY_DIR}/util/bpacking.cc" "${LIBRARY_DIR}/util/cancel.cc" "${LIBRARY_DIR}/util/compression.cc" diff --git a/docker/test/integration/runner/compose/docker_compose_mongo_secure.yml b/docker/test/integration/runner/compose/docker_compose_mongo_secure.yml new file mode 100644 index 00000000000..5d283cfc343 --- /dev/null +++ b/docker/test/integration/runner/compose/docker_compose_mongo_secure.yml @@ -0,0 +1,13 @@ +version: '2.3' +services: + mongo1: + image: mongo:3.6 + restart: always + environment: + MONGO_INITDB_ROOT_USERNAME: root + MONGO_INITDB_ROOT_PASSWORD: clickhouse + volumes: + - ${MONGO_CONFIG_PATH}:/mongo/ + ports: + - ${MONGO_EXTERNAL_PORT}:${MONGO_INTERNAL_PORT} + command: --config /mongo/mongo_secure.conf --profile=2 --verbose diff --git a/docker/test/performance-comparison/perf.py b/docker/test/performance-comparison/perf.py index 9628c512e83..a6e7e397e32 100755 --- a/docker/test/performance-comparison/perf.py +++ b/docker/test/performance-comparison/perf.py @@ -183,6 +183,10 @@ for conn_index, c in enumerate(all_connections): # requires clickhouse-driver >= 1.1.5 to accept arbitrary new settings # (https://github.com/mymarilyn/clickhouse-driver/pull/142) c.settings[s.tag] = s.text + # We have to perform a query to make sure the settings work. Otherwise an + # unknown setting will lead to failing precondition check, and we will skip + # the test, which is wrong. + c.execute("select 1") reportStageEnd('settings') diff --git a/docs/en/engines/table-engines/integrations/mongodb.md b/docs/en/engines/table-engines/integrations/mongodb.md index cc9b33192b5..9839893d4e8 100644 --- a/docs/en/engines/table-engines/integrations/mongodb.md +++ b/docs/en/engines/table-engines/integrations/mongodb.md @@ -15,7 +15,7 @@ CREATE TABLE [IF NOT EXISTS] [db.]table_name name1 [type1], name2 [type2], ... -) ENGINE = MongoDB(host:port, database, collection, user, password); +) ENGINE = MongoDB(host:port, database, collection, user, password [, options]); ``` **Engine Parameters** @@ -30,9 +30,11 @@ CREATE TABLE [IF NOT EXISTS] [db.]table_name - `password` — User password. +- `options` — MongoDB connection string options (optional parameter). + ## Usage Example {#usage-example} -Table in ClickHouse which allows to read data from MongoDB collection: +Create a table in ClickHouse which allows to read data from MongoDB collection: ``` text CREATE TABLE mongo_table @@ -42,6 +44,16 @@ CREATE TABLE mongo_table ) ENGINE = MongoDB('mongo1:27017', 'test', 'simple_table', 'testuser', 'clickhouse'); ``` +To read from an SSL secured MongoDB server: + +``` text +CREATE TABLE mongo_table_ssl +( + key UInt64, + data String +) ENGINE = MongoDB('mongo2:27017', 'test', 'simple_table', 'testuser', 'clickhouse', 'ssl=true'); +``` + Query: ``` sql diff --git a/docs/en/interfaces/third-party/gui.md b/docs/en/interfaces/third-party/gui.md index fffe0c87a53..2d7f3a24011 100644 --- a/docs/en/interfaces/third-party/gui.md +++ b/docs/en/interfaces/third-party/gui.md @@ -84,6 +84,8 @@ Features: - Table data preview. - Full-text search. +By default, DBeaver does not connect using a session (the CLI for example does). If you require session support (for example to set settings for your session), edit the driver connection properties and set session_id to a random string (it uses the http connection under the hood). Then you can use any setting from the query window + ### clickhouse-cli {#clickhouse-cli} [clickhouse-cli](https://github.com/hatarist/clickhouse-cli) is an alternative command-line client for ClickHouse, written in Python 3. diff --git a/docs/en/operations/settings/settings-users.md b/docs/en/operations/settings/settings-users.md index ee834dca98a..2c8315ad069 100644 --- a/docs/en/operations/settings/settings-users.md +++ b/docs/en/operations/settings/settings-users.md @@ -28,7 +28,7 @@ Structure of the `users` section: profile_name default - + default diff --git a/docs/en/operations/system-tables/replicas.md b/docs/en/operations/system-tables/replicas.md index 5a6ec54723b..e2cc607f6d8 100644 --- a/docs/en/operations/system-tables/replicas.md +++ b/docs/en/operations/system-tables/replicas.md @@ -82,6 +82,7 @@ The next 4 columns have a non-zero value only where there is an active session w - `absolute_delay` (`UInt64`) - How big lag in seconds the current replica has. - `total_replicas` (`UInt8`) - The total number of known replicas of this table. - `active_replicas` (`UInt8`) - The number of replicas of this table that have a session in ZooKeeper (i.e., the number of functioning replicas). +- `replica_is_active` ([Map(String, UInt8)](../../sql-reference/data-types/map.md)) — Map between replica name and is replica active. If you request all the columns, the table may work a bit slowly, since several reads from ZooKeeper are made for each row. If you do not request the last 4 columns (log_max_index, log_pointer, total_replicas, active_replicas), the table works quickly. diff --git a/docs/en/sql-reference/dictionaries/external-dictionaries/external-dicts-dict-layout.md b/docs/en/sql-reference/dictionaries/external-dictionaries/external-dicts-dict-layout.md index c69dc4224e6..ffa0fd6f29e 100644 --- a/docs/en/sql-reference/dictionaries/external-dictionaries/external-dicts-dict-layout.md +++ b/docs/en/sql-reference/dictionaries/external-dictionaries/external-dicts-dict-layout.md @@ -275,9 +275,13 @@ The dictionary is stored in a cache that has a fixed number of cells. These cell When searching for a dictionary, the cache is searched first. For each block of data, all keys that are not found in the cache or are outdated are requested from the source using `SELECT attrs... FROM db.table WHERE id IN (k1, k2, ...)`. The received data is then written to the cache. -For cache dictionaries, the expiration [lifetime](../../../sql-reference/dictionaries/external-dictionaries/external-dicts-dict-lifetime.md) of data in the cache can be set. If more time than `lifetime` has passed since loading the data in a cell, the cell’s value is not used, and it is re-requested the next time it needs to be used. +If keys are not found in dictionary, then update cache task is created and added into update queue. Update queue properties can be controlled with settings `max_update_queue_size`, `update_queue_push_timeout_milliseconds`, `query_wait_timeout_milliseconds`, `max_threads_for_updates`. + +For cache dictionaries, the expiration [lifetime](../../../sql-reference/dictionaries/external-dictionaries/external-dicts-dict-lifetime.md) of data in the cache can be set. If more time than `lifetime` has passed since loading the data in a cell, the cell’s value is not used and key becomes expired, and it is re-requested the next time it needs to be used this behaviour can be configured with setting `allow_read_expired_keys`. This is the least effective of all the ways to store dictionaries. The speed of the cache depends strongly on correct settings and the usage scenario. A cache type dictionary performs well only when the hit rates are high enough (recommended 99% and higher). You can view the average hit rate in the `system.dictionaries` table. +If setting `allow_read_expired_keys` is set to 1, by default 0. Then dictionary can support asynchronous updates. If a client requests keys and all of them are in cache, but some of them are expired, then dictionary will return expired keys for a client and request them asynchronously from the source. + To improve cache performance, use a subquery with `LIMIT`, and call the function with the dictionary externally. Supported [sources](../../../sql-reference/dictionaries/external-dictionaries/external-dicts-dict-sources.md): MySQL, ClickHouse, executable, HTTP. @@ -289,6 +293,16 @@ Example of settings: 1000000000 + + 0 + + 100000 + + 10 + + 60000 + + 4 ``` @@ -315,7 +329,7 @@ This type of storage is for use with composite [keys](../../../sql-reference/dic ### ssd_cache {#ssd-cache} -Similar to `cache`, but stores data on SSD and index in RAM. +Similar to `cache`, but stores data on SSD and index in RAM. All cache dictionary settings related to update queue can also be applied to SSD cache dictionaries. ``` xml diff --git a/docs/en/sql-reference/functions/nlp-functions.md b/docs/en/sql-reference/functions/nlp-functions.md index 2d5a09c0897..1654771574b 100644 --- a/docs/en/sql-reference/functions/nlp-functions.md +++ b/docs/en/sql-reference/functions/nlp-functions.md @@ -3,11 +3,14 @@ toc_priority: 67 toc_title: NLP --- -# Natural Language Processing functions {#nlp-functions} +# [experimental] Natural Language Processing functions {#nlp-functions} + +!!! warning "Warning" + This is an experimental feature that is currently in development and is not ready for general use. It will change in unpredictable backwards-incompatible ways in future releases. Set `allow_experimental_nlp_functions = 1` to enable it. ## stem {#stem} -Performs stemming on a previously tokenized text. +Performs stemming on a given word. **Syntax** @@ -38,7 +41,7 @@ Result: ## lemmatize {#lemmatize} -Performs lemmatization on a given word. +Performs lemmatization on a given word. Needs dictionaries to operate, which can be obtained [here](https://github.com/vpodpecan/lemmagen3/tree/master/src/lemmagen3/models). **Syntax** @@ -79,7 +82,11 @@ Configuration: ## synonyms {#synonyms} -Finds synonyms to a given word. +Finds synonyms to a given word. There are two types of synonym extensions: `plain` and `wordnet`. + +With the `plain` extension type we need to provide a path to a simple text file, where each line corresponds to a certain synonym set. Words in this line must be separated with space or tab characters. + +With the `wordnet` extension type we need to provide a path to a directory with WordNet thesaurus in it. Thesaurus must contain a WordNet sense index. **Syntax** @@ -89,7 +96,7 @@ synonyms('extension_name', word) **Arguments** -- `extension_name` — Name of the extention in which search will be performed. [String](../../sql-reference/data-types/string.md#string). +- `extension_name` — Name of the extension in which search will be performed. [String](../../sql-reference/data-types/string.md#string). - `word` — Word that will be searched in extension. [String](../../sql-reference/data-types/string.md#string). **Examples** @@ -122,4 +129,4 @@ Configuration: en/ -``` \ No newline at end of file +``` diff --git a/docs/en/sql-reference/statements/create/user.md b/docs/en/sql-reference/statements/create/user.md index ea275b9a2f8..dfa065f5d0a 100644 --- a/docs/en/sql-reference/statements/create/user.md +++ b/docs/en/sql-reference/statements/create/user.md @@ -15,6 +15,7 @@ CREATE USER [IF NOT EXISTS | OR REPLACE] name1 [ON CLUSTER cluster_name1] [NOT IDENTIFIED | IDENTIFIED {[WITH {no_password | plaintext_password | sha256_password | sha256_hash | double_sha1_password | double_sha1_hash}] BY {'password' | 'hash'}} | {WITH ldap SERVER 'server_name'} | {WITH kerberos [REALM 'realm']}] [HOST {LOCAL | NAME 'name' | REGEXP 'name_regexp' | IP 'address' | LIKE 'pattern'} [,...] | ANY | NONE] [DEFAULT ROLE role [,...]] + [DEFAULT DATABASE database | NONE] [GRANTEES {user | role | ANY | NONE} [,...] [EXCEPT {user | role} [,...]]] [SETTINGS variable [= value] [MIN [=] min_value] [MAX [=] max_value] [READONLY | WRITABLE] | PROFILE 'profile_name'] [,...] ``` diff --git a/docs/en/sql-reference/statements/select/order-by.md b/docs/en/sql-reference/statements/select/order-by.md index 37cea4668e6..156f68935b5 100644 --- a/docs/en/sql-reference/statements/select/order-by.md +++ b/docs/en/sql-reference/statements/select/order-by.md @@ -274,28 +274,28 @@ This modifier also can be combined with [LIMIT … WITH TIES modifier](../../../ `WITH FILL` modifier can be set after `ORDER BY expr` with optional `FROM expr`, `TO expr` and `STEP expr` parameters. All missed values of `expr` column will be filled sequentially and other columns will be filled as defaults. -Use following syntax for filling multiple columns add `WITH FILL` modifier with optional parameters after each field name in `ORDER BY` section. +To fill multiple columns, add `WITH FILL` modifier with optional parameters after each field name in `ORDER BY` section. ``` sql ORDER BY expr [WITH FILL] [FROM const_expr] [TO const_expr] [STEP const_numeric_expr], ... exprN [WITH FILL] [FROM expr] [TO expr] [STEP numeric_expr] ``` -`WITH FILL` can be applied only for fields with Numeric (all kind of float, decimal, int) or Date/DateTime types. +`WITH FILL` can be applied for fields with Numeric (all kinds of float, decimal, int) or Date/DateTime types. When applied for `String` fields, missed values are filled with empty strings. When `FROM const_expr` not defined sequence of filling use minimal `expr` field value from `ORDER BY`. When `TO const_expr` not defined sequence of filling use maximum `expr` field value from `ORDER BY`. When `STEP const_numeric_expr` defined then `const_numeric_expr` interprets `as is` for numeric types as `days` for Date type and as `seconds` for DateTime type. When `STEP const_numeric_expr` omitted then sequence of filling use `1.0` for numeric type, `1 day` for Date type and `1 second` for DateTime type. -For example, the following query +Example of a query without `WITH FILL`: ``` sql SELECT n, source FROM ( SELECT toFloat32(number % 10) AS n, 'original' AS source FROM numbers(10) WHERE number % 3 = 1 -) ORDER BY n +) ORDER BY n; ``` -returns +Result: ``` text ┌─n─┬─source───┐ @@ -305,16 +305,16 @@ returns └───┴──────────┘ ``` -but after apply `WITH FILL` modifier +Same query after applying `WITH FILL` modifier: ``` sql SELECT n, source FROM ( SELECT toFloat32(number % 10) AS n, 'original' AS source FROM numbers(10) WHERE number % 3 = 1 -) ORDER BY n WITH FILL FROM 0 TO 5.51 STEP 0.5 +) ORDER BY n WITH FILL FROM 0 TO 5.51 STEP 0.5; ``` -returns +Result: ``` text ┌───n─┬─source───┐ @@ -334,7 +334,7 @@ returns └─────┴──────────┘ ``` -For the case when we have multiple fields `ORDER BY field2 WITH FILL, field1 WITH FILL` order of filling will follow the order of fields in `ORDER BY` clause. +For the case with multiple fields `ORDER BY field2 WITH FILL, field1 WITH FILL` order of filling will follow the order of fields in the `ORDER BY` clause. Example: @@ -350,7 +350,7 @@ ORDER BY d1 WITH FILL STEP 5; ``` -returns +Result: ``` text ┌───d1───────┬───d2───────┬─source───┐ @@ -364,9 +364,9 @@ returns └────────────┴────────────┴──────────┘ ``` -Field `d1` does not fill and use default value cause we do not have repeated values for `d2` value, and sequence for `d1` can’t be properly calculated. +Field `d1` does not fill in and use the default value cause we do not have repeated values for `d2` value, and the sequence for `d1` can’t be properly calculated. -The following query with a changed field in `ORDER BY` +The following query with the changed field in `ORDER BY`: ``` sql SELECT @@ -380,7 +380,7 @@ ORDER BY d2 WITH FILL; ``` -returns +Result: ``` text ┌───d1───────┬───d2───────┬─source───┐ diff --git a/docs/en/sql-reference/window-functions/index.md b/docs/en/sql-reference/window-functions/index.md index dcab019c9d5..e62808a46bd 100644 --- a/docs/en/sql-reference/window-functions/index.md +++ b/docs/en/sql-reference/window-functions/index.md @@ -5,9 +5,6 @@ toc_title: Window Functions # [experimental] Window Functions -!!! warning "Warning" - This is an experimental feature that is currently in development and is not ready for general use. It will change in unpredictable backwards-incompatible ways in the future releases. Set `allow_experimental_window_functions = 1` to enable it. - ClickHouse supports the standard grammar for defining windows and window functions. The following features are currently supported: | Feature | Support or workaround | diff --git a/docs/ru/engines/table-engines/index.md b/docs/ru/engines/table-engines/index.md index b17b2124250..1636efc5112 100644 --- a/docs/ru/engines/table-engines/index.md +++ b/docs/ru/engines/table-engines/index.md @@ -87,7 +87,7 @@ toc_title: "Введение" Виртуальный столбец — это неотъемлемый атрибут движка таблиц, определенный в исходном коде движка. -Виртуальные столбцы не надо указывать в запросе `CREATE TABLE` и их не отображаются в результатах запросов `SHOW CREATE TABLE` и `DESCRIBE TABLE`. Также виртуальные столбцы доступны только для чтения, поэтому вы не можете вставлять в них данные. +Виртуальные столбцы не надо указывать в запросе `CREATE TABLE` и они не отображаются в результатах запросов `SHOW CREATE TABLE` и `DESCRIBE TABLE`. Также виртуальные столбцы доступны только для чтения, поэтому вы не можете вставлять в них данные. Чтобы получить данные из виртуального столбца, необходимо указать его название в запросе `SELECT`. `SELECT *` не отображает данные из виртуальных столбцов. diff --git a/docs/ru/interfaces/third-party/gui.md b/docs/ru/interfaces/third-party/gui.md index dc96c32e996..9cb28a2c9a2 100644 --- a/docs/ru/interfaces/third-party/gui.md +++ b/docs/ru/interfaces/third-party/gui.md @@ -111,7 +111,7 @@ toc_title: "Визуальные интерфейсы от сторонних р ### DataGrip {#datagrip} -[DataGrip](https://www.jetbrains.com/datagrip/) — это IDE для баз данных о JetBrains с выделенной поддержкой ClickHouse. Он также встроен в другие инструменты на основе IntelliJ: PyCharm, IntelliJ IDEA, GoLand, PhpStorm и другие. +[DataGrip](https://www.jetbrains.com/datagrip/) — это IDE для баз данных от JetBrains с выделенной поддержкой ClickHouse. Он также встроен в другие инструменты на основе IntelliJ: PyCharm, IntelliJ IDEA, GoLand, PhpStorm и другие. Основные возможности: diff --git a/docs/ru/sql-reference/functions/nlp-functions.md b/docs/ru/sql-reference/functions/nlp-functions.md index 582b5c93b93..58c4eb86e35 100644 --- a/docs/ru/sql-reference/functions/nlp-functions.md +++ b/docs/ru/sql-reference/functions/nlp-functions.md @@ -3,7 +3,10 @@ toc_priority: 67 toc_title: NLP --- -# Функции для работы с ествественным языком {#nlp-functions} +# [экспериментально] Функции для работы с ествественным языком {#nlp-functions} + +!!! warning "Предупреждение" + Сейчас использование функций для работы с ествественным языком является экспериментальной возможностью. Чтобы использовать данные функции, включите настройку `allow_experimental_nlp_functions = 1`. ## stem {#stem} @@ -38,7 +41,7 @@ Result: ## lemmatize {#lemmatize} -Данная функция проводит лемматизацию для заданного слова. +Данная функция проводит лемматизацию для заданного слова. Для работы лемматизатора необходимы словари, которые можно найти [здесь](https://github.com/vpodpecan/lemmagen3/tree/master/src/lemmagen3/models). **Синтаксис** @@ -79,7 +82,11 @@ SELECT lemmatize('en', 'wolves'); ## synonyms {#synonyms} -Находит синонимы к заданному слову. +Находит синонимы к заданному слову. Представлены два типа расширений словарей: `plain` и `wordnet`. + +Для работы расширения типа `plain` необходимо указать путь до простого текстового файла, где каждая строка соотвествует одному набору синонимов. Слова в данной строке должны быть разделены с помощью пробела или знака табуляции. + +Для работы расширения типа `plain` необходимо указать путь до WordNet тезауруса. Тезаурус должен содержать WordNet sense index. **Синтаксис** diff --git a/docs/ru/sql-reference/statements/select/order-by.md b/docs/ru/sql-reference/statements/select/order-by.md index fbbf8c842e0..d7d2e9c7574 100644 --- a/docs/ru/sql-reference/statements/select/order-by.md +++ b/docs/ru/sql-reference/statements/select/order-by.md @@ -271,8 +271,8 @@ SELECT * FROM collate_test ORDER BY s ASC COLLATE 'en'; Этот модификатор также может быть скобинирован с модификатором [LIMIT ... WITH TIES](../../../sql-reference/statements/select/limit.md#limit-with-ties) -`WITH FILL` модификатор может быть установлен после `ORDER BY expr` с опциональными параметрами `FROM expr`, `TO expr` и `STEP expr`. -Все пропущенные значнеия для колонки `expr` будут заполненые значениями соответсвующими предполагаемой последовательности значений колонки, другие колонки будут заполнены значенями по умолчанию. +Модификатор `WITH FILL` может быть установлен после `ORDER BY expr` с опциональными параметрами `FROM expr`, `TO expr` и `STEP expr`. +Все пропущенные значения для колонки `expr` будут заполнены значениями, соответствующими предполагаемой последовательности значений колонки, другие колонки будут заполнены значениями по умолчанию. Используйте следующую конструкцию для заполнения нескольких колонок с модификатором `WITH FILL` с необязательными параметрами после каждого имени поля в секции `ORDER BY`. @@ -280,22 +280,22 @@ SELECT * FROM collate_test ORDER BY s ASC COLLATE 'en'; ORDER BY expr [WITH FILL] [FROM const_expr] [TO const_expr] [STEP const_numeric_expr], ... exprN [WITH FILL] [FROM expr] [TO expr] [STEP numeric_expr] ``` -`WITH FILL` может быть применене только к полям с числовыми (все разновидности float, int, decimal) или временными (все разновидности Date, DateTime) типами. +`WITH FILL` может быть применен к полям с числовыми (все разновидности float, int, decimal) или временными (все разновидности Date, DateTime) типами. В случае применения к полям типа `String` недостающие значения заполняются пустой строкой. Когда не определен `FROM const_expr`, последовательность заполнения использует минимальное значение поля `expr` из `ORDER BY`. Когда не определен `TO const_expr`, последовательность заполнения использует максимальное значение поля `expr` из `ORDER BY`. -Когда `STEP const_numeric_expr` определен, тогда `const_numeric_expr` интерпретируется `как есть` для числовых типов, как `дни` для типа Date и как `секунды` для типа DateTime. +Когда `STEP const_numeric_expr` определен, `const_numeric_expr` интерпретируется "как есть" для числовых типов, как "дни" для типа `Date` и как "секунды" для типа `DateTime`. + Когда `STEP const_numeric_expr` не указан, тогда используется `1.0` для числовых типов, `1 день` для типа Date и `1 секунда` для типа DateTime. - -Для примера, следующий запрос +Пример запроса без использования `WITH FILL`: ```sql SELECT n, source FROM ( SELECT toFloat32(number % 10) AS n, 'original' AS source FROM numbers(10) WHERE number % 3 = 1 -) ORDER BY n +) ORDER BY n; ``` -возвращает +Результат: ```text ┌─n─┬─source───┐ │ 1 │ original │ @@ -304,7 +304,7 @@ SELECT n, source FROM ( └───┴──────────┘ ``` -но после применения модификатора `WITH FILL` +Тот же запрос после применения модификатора `WITH FILL`: ```sql SELECT n, source FROM ( SELECT toFloat32(number % 10) AS n, 'original' AS source @@ -312,7 +312,8 @@ SELECT n, source FROM ( ) ORDER BY n WITH FILL FROM 0 TO 5.51 STEP 0.5 ``` -возвращает +Результат: + ```text ┌───n─┬─source───┐ │ 0 │ │ @@ -331,13 +332,13 @@ SELECT n, source FROM ( └─────┴──────────┘ ``` -Для случая когда у нас есть несколько полей `ORDER BY field2 WITH FILL, field1 WITH FILL` порядок заполнения будет следовать порядку полей в секции `ORDER BY`. +Для случая с несколькими полями `ORDER BY field2 WITH FILL, field1 WITH FILL` порядок заполнения будет соответствовать порядку полей в секции `ORDER BY`. Пример: -```sql -SELECT - toDate((number * 10) * 86400) AS d1, - toDate(number * 86400) AS d2, +```sql +SELECT + toDate((number * 10) * 86400) AS d1, + toDate(number * 86400) AS d2, 'original' AS source FROM numbers(10) WHERE (number % 3) = 1 @@ -346,7 +347,7 @@ ORDER BY d1 WITH FILL STEP 5; ``` -возвращает +Результат: ```text ┌───d1───────┬───d2───────┬─source───┐ │ 1970-01-11 │ 1970-01-02 │ original │ @@ -359,9 +360,9 @@ ORDER BY └────────────┴────────────┴──────────┘ ``` -Поле `d1` не заполняет и используется значение по умолчанию поскольку у нас нет повторяющихся значения для `d2` поэтому мы не можем правильно рассчитать последователность заполнения для`d1`. +Поле `d1` не заполняется и использует значение по умолчанию. Поскольку у нас нет повторяющихся значений для `d2`, мы не можем правильно рассчитать последователность заполнения для `d1`. -Cледующий запрос (с измененым порядком в ORDER BY) +Cледующий запрос (с измененым порядком в ORDER BY): ```sql SELECT toDate((number * 10) * 86400) AS d1, @@ -374,7 +375,7 @@ ORDER BY d2 WITH FILL; ``` -возвращает +Результат: ```text ┌───d1───────┬───d2───────┬─source───┐ │ 1970-01-11 │ 1970-01-02 │ original │ diff --git a/programs/client/Client.cpp b/programs/client/Client.cpp index c5e43449557..5b204751ba0 100644 --- a/programs/client/Client.cpp +++ b/programs/client/Client.cpp @@ -197,7 +197,7 @@ private: std::unique_ptr pager_cmd; /// The user can specify to redirect query output to a file. - std::optional out_file_buf; + std::unique_ptr out_file_buf; BlockOutputStreamPtr block_out_stream; /// The user could specify special file for server logs (stderr by default) @@ -2238,8 +2238,11 @@ private: const auto & out_file_node = query_with_output->out_file->as(); const auto & out_file = out_file_node.value.safeGet(); - out_file_buf.emplace(out_file, DBMS_DEFAULT_BUFFER_SIZE, O_WRONLY | O_EXCL | O_CREAT); - out_buf = &*out_file_buf; + out_file_buf = wrapWriteBufferWithCompressionMethod( + std::make_unique(out_file, DBMS_DEFAULT_BUFFER_SIZE, O_WRONLY | O_EXCL | O_CREAT), + chooseCompressionMethod(out_file, ""), + /* compression level = */ 3 + ); // We are writing to file, so default format is the same as in non-interactive mode. if (is_interactive && is_default_format) @@ -2259,9 +2262,9 @@ private: /// It is not clear how to write progress with parallel formatting. It may increase code complexity significantly. if (!need_render_progress) - block_out_stream = context->getOutputStreamParallelIfPossible(current_format, *out_buf, block); + block_out_stream = context->getOutputStreamParallelIfPossible(current_format, out_file_buf ? *out_file_buf : *out_buf, block); else - block_out_stream = context->getOutputStream(current_format, *out_buf, block); + block_out_stream = context->getOutputStream(current_format, out_file_buf ? *out_file_buf : *out_buf, block); block_out_stream->writePrefix(); } diff --git a/programs/library-bridge/HandlerFactory.cpp b/programs/library-bridge/HandlerFactory.cpp index 9f53a24156f..43087082c46 100644 --- a/programs/library-bridge/HandlerFactory.cpp +++ b/programs/library-bridge/HandlerFactory.cpp @@ -12,8 +12,8 @@ namespace DB Poco::URI uri{request.getURI()}; LOG_DEBUG(log, "Request URI: {}", uri.toString()); - if (uri == "/ping" && request.getMethod() == Poco::Net::HTTPRequest::HTTP_GET) - return std::make_unique(keep_alive_timeout); + if (request.getMethod() == Poco::Net::HTTPRequest::HTTP_GET) + return std::make_unique(keep_alive_timeout, getContext()); if (request.getMethod() == Poco::Net::HTTPRequest::HTTP_POST) return std::make_unique(keep_alive_timeout, getContext()); diff --git a/programs/library-bridge/Handlers.cpp b/programs/library-bridge/Handlers.cpp index 89219d1b910..2b6d0057bb2 100644 --- a/programs/library-bridge/Handlers.cpp +++ b/programs/library-bridge/Handlers.cpp @@ -17,8 +17,24 @@ namespace DB { + +namespace ErrorCodes +{ + extern const int BAD_REQUEST_PARAMETER; +} + namespace { + void processError(HTTPServerResponse & response, const std::string & message) + { + response.setStatusAndReason(HTTPResponse::HTTP_INTERNAL_SERVER_ERROR); + + if (!response.sent()) + *response.send() << message << std::endl; + + LOG_WARNING(&Poco::Logger::get("LibraryBridge"), message); + } + std::shared_ptr parseColumns(std::string && column_string) { auto sample_block = std::make_shared(); @@ -30,9 +46,8 @@ namespace return sample_block; } - std::vector parseIdsFromBinary(const std::string & ids_string) + std::vector parseIdsFromBinary(ReadBuffer & buf) { - ReadBufferFromString buf(ids_string); std::vector ids; readVectorBinary(ids, buf); return ids; @@ -67,13 +82,36 @@ void LibraryRequestHandler::handleRequest(HTTPServerRequest & request, HTTPServe std::string method = params.get("method"); std::string dictionary_id = params.get("dictionary_id"); - LOG_TRACE(log, "Library method: '{}', dictionary id: {}", method, dictionary_id); + LOG_TRACE(log, "Library method: '{}', dictionary id: {}", method, dictionary_id); WriteBufferFromHTTPServerResponse out(response, request.getMethod() == Poco::Net::HTTPRequest::HTTP_HEAD, keep_alive_timeout); try { - if (method == "libNew") + bool lib_new = (method == "libNew"); + if (method == "libClone") + { + if (!params.has("from_dictionary_id")) + { + processError(response, "No 'from_dictionary_id' in request URL"); + return; + } + + std::string from_dictionary_id = params.get("from_dictionary_id"); + bool cloned = false; + cloned = SharedLibraryHandlerFactory::instance().clone(from_dictionary_id, dictionary_id); + + if (cloned) + { + writeStringBinary("1", out); + } + else + { + LOG_TRACE(log, "Cannot clone from dictionary with id: {}, will call libNew instead"); + lib_new = true; + } + } + if (lib_new) { auto & read_buf = request.getStream(); params.read(read_buf); @@ -92,6 +130,8 @@ void LibraryRequestHandler::handleRequest(HTTPServerRequest & request, HTTPServe std::string library_path = params.get("library_path"); const auto & settings_string = params.get("library_settings"); + + LOG_DEBUG(log, "Parsing library settings from binary string"); std::vector library_settings = parseNamesFromBinary(settings_string); /// Needed for library dictionary @@ -102,6 +142,8 @@ void LibraryRequestHandler::handleRequest(HTTPServerRequest & request, HTTPServe } const auto & attributes_string = params.get("attributes_names"); + + LOG_DEBUG(log, "Parsing attributes names from binary string"); std::vector attributes_names = parseNamesFromBinary(attributes_string); /// Needed to parse block from binary string format @@ -140,54 +182,63 @@ void LibraryRequestHandler::handleRequest(HTTPServerRequest & request, HTTPServe SharedLibraryHandlerFactory::instance().create(dictionary_id, library_path, library_settings, sample_block_with_nulls, attributes_names); writeStringBinary("1", out); } - else if (method == "libClone") - { - if (!params.has("from_dictionary_id")) - { - processError(response, "No 'from_dictionary_id' in request URL"); - return; - } - - std::string from_dictionary_id = params.get("from_dictionary_id"); - LOG_TRACE(log, "Calling libClone from {} to {}", from_dictionary_id, dictionary_id); - SharedLibraryHandlerFactory::instance().clone(from_dictionary_id, dictionary_id); - writeStringBinary("1", out); - } else if (method == "libDelete") { - SharedLibraryHandlerFactory::instance().remove(dictionary_id); + auto deleted = SharedLibraryHandlerFactory::instance().remove(dictionary_id); + + /// Do not throw, a warning is ok. + if (!deleted) + LOG_WARNING(log, "Cannot delete library for with dictionary id: {}, because such id was not found.", dictionary_id); + writeStringBinary("1", out); } else if (method == "isModified") { auto library_handler = SharedLibraryHandlerFactory::instance().get(dictionary_id); + if (!library_handler) + throw Exception(ErrorCodes::BAD_REQUEST_PARAMETER, "Not found dictionary with id: {}", dictionary_id); + bool res = library_handler->isModified(); writeStringBinary(std::to_string(res), out); } else if (method == "supportsSelectiveLoad") { auto library_handler = SharedLibraryHandlerFactory::instance().get(dictionary_id); + if (!library_handler) + throw Exception(ErrorCodes::BAD_REQUEST_PARAMETER, "Not found dictionary with id: {}", dictionary_id); + bool res = library_handler->supportsSelectiveLoad(); writeStringBinary(std::to_string(res), out); } else if (method == "loadAll") { auto library_handler = SharedLibraryHandlerFactory::instance().get(dictionary_id); + if (!library_handler) + throw Exception(ErrorCodes::BAD_REQUEST_PARAMETER, "Not found dictionary with id: {}", dictionary_id); + const auto & sample_block = library_handler->getSampleBlock(); + LOG_DEBUG(log, "Calling loadAll() for dictionary id: {}", dictionary_id); auto input = library_handler->loadAll(); + LOG_DEBUG(log, "Started sending result data for dictionary id: {}", dictionary_id); BlockOutputStreamPtr output = FormatFactory::instance().getOutputStream(FORMAT, out, sample_block, getContext()); copyData(*input, *output); } else if (method == "loadIds") { + LOG_DEBUG(log, "Getting diciontary ids for dictionary with id: {}", dictionary_id); String ids_string; - readString(ids_string, request.getStream()); - std::vector ids = parseIdsFromBinary(ids_string); + std::vector ids = parseIdsFromBinary(request.getStream()); auto library_handler = SharedLibraryHandlerFactory::instance().get(dictionary_id); + if (!library_handler) + throw Exception(ErrorCodes::BAD_REQUEST_PARAMETER, "Not found dictionary with id: {}", dictionary_id); + const auto & sample_block = library_handler->getSampleBlock(); + LOG_DEBUG(log, "Calling loadIds() for dictionary id: {}", dictionary_id); auto input = library_handler->loadIds(ids); + + LOG_DEBUG(log, "Started sending result data for dictionary id: {}", dictionary_id); BlockOutputStreamPtr output = FormatFactory::instance().getOutputStream(FORMAT, out, sample_block, getContext()); copyData(*input, *output); } @@ -219,8 +270,14 @@ void LibraryRequestHandler::handleRequest(HTTPServerRequest & request, HTTPServe auto block = reader->read(); auto library_handler = SharedLibraryHandlerFactory::instance().get(dictionary_id); + if (!library_handler) + throw Exception(ErrorCodes::BAD_REQUEST_PARAMETER, "Not found dictionary with id: {}", dictionary_id); + const auto & sample_block = library_handler->getSampleBlock(); + LOG_DEBUG(log, "Calling loadKeys() for dictionary id: {}", dictionary_id); auto input = library_handler->loadKeys(block.getColumns()); + + LOG_DEBUG(log, "Started sending result data for dictionary id: {}", dictionary_id); BlockOutputStreamPtr output = FormatFactory::instance().getOutputStream(FORMAT, out, sample_block, getContext()); copyData(*input, *output); } @@ -228,8 +285,9 @@ void LibraryRequestHandler::handleRequest(HTTPServerRequest & request, HTTPServe catch (...) { auto message = getCurrentExceptionMessage(true); - response.setStatusAndReason(Poco::Net::HTTPResponse::HTTP_INTERNAL_SERVER_ERROR, message); // can't call process_error, because of too soon response sending + LOG_ERROR(log, "Failed to process request for dictionary_id: {}. Error: {}", dictionary_id, message); + response.setStatusAndReason(Poco::Net::HTTPResponse::HTTP_INTERNAL_SERVER_ERROR, message); // can't call process_error, because of too soon response sending try { writeStringBinary(message, out); @@ -239,8 +297,6 @@ void LibraryRequestHandler::handleRequest(HTTPServerRequest & request, HTTPServe { tryLogCurrentException(log); } - - tryLogCurrentException(log); } try @@ -254,24 +310,30 @@ void LibraryRequestHandler::handleRequest(HTTPServerRequest & request, HTTPServe } -void LibraryRequestHandler::processError(HTTPServerResponse & response, const std::string & message) -{ - response.setStatusAndReason(HTTPResponse::HTTP_INTERNAL_SERVER_ERROR); - - if (!response.sent()) - *response.send() << message << std::endl; - - LOG_WARNING(log, message); -} - - -void PingHandler::handleRequest(HTTPServerRequest & /* request */, HTTPServerResponse & response) +void LibraryExistsHandler::handleRequest(HTTPServerRequest & request, HTTPServerResponse & response) { try { + LOG_TRACE(log, "Request URI: {}", request.getURI()); + HTMLForm params(getContext()->getSettingsRef(), request); + + if (!params.has("dictionary_id")) + { + processError(response, "No 'dictionary_id' in request URL"); + return; + } + + std::string dictionary_id = params.get("dictionary_id"); + auto library_handler = SharedLibraryHandlerFactory::instance().get(dictionary_id); + String res; + if (library_handler) + res = "1"; + else + res = "0"; + setResponseDefaultHeaders(response, keep_alive_timeout); - const char * data = "Ok.\n"; - response.sendBuffer(data, strlen(data)); + LOG_TRACE(log, "Senging ping response: {} (dictionary id: {})", res, dictionary_id); + response.sendBuffer(res.data(), res.size()); } catch (...) { diff --git a/programs/library-bridge/Handlers.h b/programs/library-bridge/Handlers.h index dac61d3a735..58af24b06d1 100644 --- a/programs/library-bridge/Handlers.h +++ b/programs/library-bridge/Handlers.h @@ -22,8 +22,7 @@ class LibraryRequestHandler : public HTTPRequestHandler, WithContext public: LibraryRequestHandler( - size_t keep_alive_timeout_, - ContextPtr context_) + size_t keep_alive_timeout_, ContextPtr context_) : WithContext(context_) , log(&Poco::Logger::get("LibraryRequestHandler")) , keep_alive_timeout(keep_alive_timeout_) @@ -35,18 +34,18 @@ public: private: static constexpr inline auto FORMAT = "RowBinary"; - void processError(HTTPServerResponse & response, const std::string & message); - Poco::Logger * log; size_t keep_alive_timeout; }; -class PingHandler : public HTTPRequestHandler +class LibraryExistsHandler : public HTTPRequestHandler, WithContext { public: - explicit PingHandler(size_t keep_alive_timeout_) - : keep_alive_timeout(keep_alive_timeout_) + explicit LibraryExistsHandler(size_t keep_alive_timeout_, ContextPtr context_) + : WithContext(context_) + , keep_alive_timeout(keep_alive_timeout_) + , log(&Poco::Logger::get("LibraryRequestHandler")) { } @@ -54,6 +53,8 @@ public: private: const size_t keep_alive_timeout; + Poco::Logger * log; + }; } diff --git a/programs/library-bridge/SharedLibraryHandlerFactory.cpp b/programs/library-bridge/SharedLibraryHandlerFactory.cpp index 270e07a1046..a9358ca552a 100644 --- a/programs/library-bridge/SharedLibraryHandlerFactory.cpp +++ b/programs/library-bridge/SharedLibraryHandlerFactory.cpp @@ -4,12 +4,6 @@ namespace DB { -namespace ErrorCodes -{ - extern const int BAD_ARGUMENTS; - extern const int LOGICAL_ERROR; -} - SharedLibraryHandlerPtr SharedLibraryHandlerFactory::get(const std::string & dictionary_id) { std::lock_guard lock(mutex); @@ -18,7 +12,7 @@ SharedLibraryHandlerPtr SharedLibraryHandlerFactory::get(const std::string & dic if (library_handler != library_handlers.end()) return library_handler->second; - throw Exception(ErrorCodes::BAD_ARGUMENTS, "Not found dictionary with id: {}", dictionary_id); + return nullptr; } @@ -30,32 +24,32 @@ void SharedLibraryHandlerFactory::create( const std::vector & attributes_names) { std::lock_guard lock(mutex); - library_handlers[dictionary_id] = std::make_shared(library_path, library_settings, sample_block, attributes_names); + if (!library_handlers.count(dictionary_id)) + library_handlers.emplace(std::make_pair(dictionary_id, std::make_shared(library_path, library_settings, sample_block, attributes_names))); + else + LOG_WARNING(&Poco::Logger::get("SharedLibraryHandlerFactory"), "Library handler with dictionary id {} already exists", dictionary_id); } -void SharedLibraryHandlerFactory::clone(const std::string & from_dictionary_id, const std::string & to_dictionary_id) +bool SharedLibraryHandlerFactory::clone(const std::string & from_dictionary_id, const std::string & to_dictionary_id) { std::lock_guard lock(mutex); auto from_library_handler = library_handlers.find(from_dictionary_id); - /// This is not supposed to happen as libClone is called from copy constructor of LibraryDictionarySource - /// object, and shared library handler of from_dictionary is removed only in its destructor. - /// And if for from_dictionary there was no shared library handler, it would have received and exception in - /// its constructor, so no libClone would be made from it. if (from_library_handler == library_handlers.end()) - throw Exception(ErrorCodes::LOGICAL_ERROR, "No shared library handler found"); + return false; /// libClone method will be called in copy constructor library_handlers[to_dictionary_id] = std::make_shared(*from_library_handler->second); + return true; } -void SharedLibraryHandlerFactory::remove(const std::string & dictionary_id) +bool SharedLibraryHandlerFactory::remove(const std::string & dictionary_id) { std::lock_guard lock(mutex); /// libDelete is called in destructor. - library_handlers.erase(dictionary_id); + return library_handlers.erase(dictionary_id); } diff --git a/programs/library-bridge/SharedLibraryHandlerFactory.h b/programs/library-bridge/SharedLibraryHandlerFactory.h index 473d90618a2..115cc78ae52 100644 --- a/programs/library-bridge/SharedLibraryHandlerFactory.h +++ b/programs/library-bridge/SharedLibraryHandlerFactory.h @@ -24,9 +24,9 @@ public: const Block & sample_block, const std::vector & attributes_names); - void clone(const std::string & from_dictionary_id, const std::string & to_dictionary_id); + bool clone(const std::string & from_dictionary_id, const std::string & to_dictionary_id); - void remove(const std::string & dictionary_id); + bool remove(const std::string & dictionary_id); private: /// map: dict_id -> sharedLibraryHandler diff --git a/programs/server/play.html b/programs/server/play.html index 7b13807f2d9..503bb92d03e 100644 --- a/programs/server/play.html +++ b/programs/server/play.html @@ -361,23 +361,22 @@ function postImpl(posted_request_num, query) { - /// TODO: Check if URL already contains query string (append parameters). + const user = document.getElementById('user').value; + const password = document.getElementById('password').value; - let user = document.getElementById('user').value; - let password = document.getElementById('password').value; + const server_address = document.getElementById('url').value; - let server_address = document.getElementById('url').value; - - let url = server_address + + const url = server_address + + (server_address.indexOf('?') >= 0 ? '&' : '?') + /// Ask server to allow cross-domain requests. - '?add_http_cors_header=1' + + 'add_http_cors_header=1' + '&user=' + encodeURIComponent(user) + '&password=' + encodeURIComponent(password) + '&default_format=JSONCompact' + /// Safety settings to prevent results that browser cannot display. '&max_result_rows=1000&max_result_bytes=10000000&result_overflow_mode=break'; - let xhr = new XMLHttpRequest; + const xhr = new XMLHttpRequest; xhr.open('POST', url, true); @@ -391,12 +390,12 @@ /// The query is saved in browser history (in state JSON object) /// as well as in URL fragment identifier. if (query != previous_query) { - let state = { + const state = { query: query, status: this.status, response: this.response.length > 100000 ? null : this.response /// Lower than the browser's limit. }; - let title = "ClickHouse Query: " + query; + const title = "ClickHouse Query: " + query; let history_url = window.location.pathname + '?user=' + encodeURIComponent(user); if (server_address != location.origin) { diff --git a/src/Bridge/IBridgeHelper.cpp b/src/Bridge/IBridgeHelper.cpp index b6f3446d0a6..5c884a2ca3d 100644 --- a/src/Bridge/IBridgeHelper.cpp +++ b/src/Bridge/IBridgeHelper.cpp @@ -33,24 +33,9 @@ Poco::URI IBridgeHelper::getPingURI() const } -bool IBridgeHelper::checkBridgeIsRunning() const +void IBridgeHelper::startBridgeSync() { - try - { - ReadWriteBufferFromHTTP buf( - getPingURI(), Poco::Net::HTTPRequest::HTTP_GET, {}, ConnectionTimeouts::getHTTPTimeouts(getContext())); - return checkString(PING_OK_ANSWER, buf); - } - catch (...) - { - return false; - } -} - - -void IBridgeHelper::startBridgeSync() const -{ - if (!checkBridgeIsRunning()) + if (!bridgeHandShake()) { LOG_TRACE(getLog(), "{} is not running, will try to start it", serviceAlias()); startBridge(startBridgeCommand()); @@ -64,7 +49,7 @@ void IBridgeHelper::startBridgeSync() const ++counter; LOG_TRACE(getLog(), "Checking {} is running, try {}", serviceAlias(), counter); - if (checkBridgeIsRunning()) + if (bridgeHandShake()) { started = true; break; @@ -81,7 +66,7 @@ void IBridgeHelper::startBridgeSync() const } -std::unique_ptr IBridgeHelper::startBridgeCommand() const +std::unique_ptr IBridgeHelper::startBridgeCommand() { if (startBridgeManually()) throw Exception(serviceAlias() + " is not running. Please, start it manually", ErrorCodes::EXTERNAL_SERVER_IS_NOT_RESPONDING); diff --git a/src/Bridge/IBridgeHelper.h b/src/Bridge/IBridgeHelper.h index caaf031b7d8..0537658663d 100644 --- a/src/Bridge/IBridgeHelper.h +++ b/src/Bridge/IBridgeHelper.h @@ -28,16 +28,19 @@ public: static const inline std::string MAIN_METHOD = Poco::Net::HTTPRequest::HTTP_POST; explicit IBridgeHelper(ContextPtr context_) : WithContext(context_) {} - virtual ~IBridgeHelper() = default; - void startBridgeSync() const; + virtual ~IBridgeHelper() = default; Poco::URI getMainURI() const; Poco::URI getPingURI() const; + void startBridgeSync(); protected: + /// Check bridge is running. Can also check something else in the mean time. + virtual bool bridgeHandShake() = 0; + /// clickhouse-odbc-bridge, clickhouse-library-bridge virtual String serviceAlias() const = 0; @@ -61,9 +64,7 @@ protected: private: - bool checkBridgeIsRunning() const; - - std::unique_ptr startBridgeCommand() const; + std::unique_ptr startBridgeCommand(); }; } diff --git a/src/Bridge/LibraryBridgeHelper.cpp b/src/Bridge/LibraryBridgeHelper.cpp index b13be0aba29..95a96074be0 100644 --- a/src/Bridge/LibraryBridgeHelper.cpp +++ b/src/Bridge/LibraryBridgeHelper.cpp @@ -1,6 +1,5 @@ #include "LibraryBridgeHelper.h" -#include #include #include #include @@ -8,6 +7,8 @@ #include #include #include +#include +#include #include #include #include @@ -20,16 +21,25 @@ namespace DB { +namespace ErrorCodes +{ + extern const int EXTERNAL_LIBRARY_ERROR; + extern const int LOGICAL_ERROR; +} + LibraryBridgeHelper::LibraryBridgeHelper( ContextPtr context_, const Block & sample_block_, - const Field & dictionary_id_) + const Field & dictionary_id_, + const LibraryInitData & library_data_) : IBridgeHelper(context_->getGlobalContext()) , log(&Poco::Logger::get("LibraryBridgeHelper")) , sample_block(sample_block_) , config(context_->getConfigRef()) , http_timeout(context_->getGlobalContext()->getSettingsRef().http_receive_timeout.value) + , library_data(library_data_) , dictionary_id(dictionary_id_) + , http_timeouts(ConnectionTimeouts::getHTTPTimeouts(context_)) { bridge_port = config.getUInt("library_bridge.port", DEFAULT_PORT); bridge_host = config.getString("library_bridge.host", DEFAULT_HOST); @@ -61,26 +71,91 @@ void LibraryBridgeHelper::startBridge(std::unique_ptr cmd) const } -bool LibraryBridgeHelper::initLibrary(const std::string & library_path, const std::string library_settings, const std::string attributes_names) +bool LibraryBridgeHelper::bridgeHandShake() { - startBridgeSync(); - auto uri = createRequestURI(LIB_NEW_METHOD); + String result; + try + { + ReadWriteBufferFromHTTP buf(createRequestURI(PING), Poco::Net::HTTPRequest::HTTP_GET, {}, http_timeouts); + readString(result, buf); + } + catch (...) + { + return false; + } + /* + * When pinging bridge we also pass current dicionary_id. The bridge will check if there is such + * dictionary. It is possible that such dictionary_id is not present only in two cases: + * 1. It is dictionary source creation and initialization of library handler on bridge side did not happen yet. + * 2. Bridge crashed or restarted for some reason while server did not. + **/ + if (result.size() != 1) + throw Exception(ErrorCodes::LOGICAL_ERROR, "Unexpected message from library bridge: {}. Check bridge and server have the same version.", result); + + UInt8 dictionary_id_exists; + auto parsed = tryParse(dictionary_id_exists, result); + if (!parsed || (dictionary_id_exists != 0 && dictionary_id_exists != 1)) + throw Exception(ErrorCodes::LOGICAL_ERROR, "Unexpected message from library bridge: {} ({}). Check bridge and server have the same version.", + result, parsed ? toString(dictionary_id_exists) : "failed to parse"); + + LOG_TRACE(log, "dictionary_id: {}, dictionary_id_exists on bridge side: {}, library confirmed to be initialized on server side: {}", + toString(dictionary_id), toString(dictionary_id_exists), library_initialized); + + if (dictionary_id_exists && !library_initialized) + throw Exception(ErrorCodes::LOGICAL_ERROR, "Library was not initialized, but bridge responded to already have dictionary id: {}", dictionary_id); + + /// Here we want to say bridge to recreate a new library handler for current dictionary, + /// because it responded to have lost it, but we know that it has already been created. (It is a direct result of bridge crash). + if (!dictionary_id_exists && library_initialized) + { + LOG_WARNING(log, "Library bridge does not have library handler with dictionaty id: {}. It will be reinitialized.", dictionary_id); + bool reinitialized = false; + try + { + auto uri = createRequestURI(LIB_NEW_METHOD); + reinitialized = executeRequest(uri, getInitLibraryCallback()); + } + catch (...) + { + tryLogCurrentException(log); + return false; + } + + if (!reinitialized) + throw Exception(ErrorCodes::EXTERNAL_LIBRARY_ERROR, + "Failed to reinitialize library handler on bridge side for dictionary with id: {}", dictionary_id); + } + + return true; +} + + +ReadWriteBufferFromHTTP::OutStreamCallback LibraryBridgeHelper::getInitLibraryCallback() const +{ /// Sample block must contain null values WriteBufferFromOwnString out; auto output_stream = getContext()->getOutputStream(LibraryBridgeHelper::DEFAULT_FORMAT, out, sample_block); formatBlock(output_stream, sample_block); auto block_string = out.str(); - auto out_stream_callback = [library_path, library_settings, attributes_names, block_string, this](std::ostream & os) + return [block_string, this](std::ostream & os) { - os << "library_path=" << escapeForFileName(library_path) << "&"; - os << "library_settings=" << escapeForFileName(library_settings) << "&"; - os << "attributes_names=" << escapeForFileName(attributes_names) << "&"; + os << "library_path=" << escapeForFileName(library_data.library_path) << "&"; + os << "library_settings=" << escapeForFileName(library_data.library_settings) << "&"; + os << "attributes_names=" << escapeForFileName(library_data.dict_attributes) << "&"; os << "sample_block=" << escapeForFileName(sample_block.getNamesAndTypesList().toString()) << "&"; os << "null_values=" << escapeForFileName(block_string); }; - return executeRequest(uri, out_stream_callback); +} + + +bool LibraryBridgeHelper::initLibrary() +{ + startBridgeSync(); + auto uri = createRequestURI(LIB_NEW_METHOD); + library_initialized = executeRequest(uri, getInitLibraryCallback()); + return library_initialized; } @@ -89,15 +164,23 @@ bool LibraryBridgeHelper::cloneLibrary(const Field & other_dictionary_id) startBridgeSync(); auto uri = createRequestURI(LIB_CLONE_METHOD); uri.addQueryParameter("from_dictionary_id", toString(other_dictionary_id)); - return executeRequest(uri); + /// We also pass initialization settings in order to create a library handler + /// in case from_dictionary_id does not exist in bridge side (possible in case of bridge crash). + library_initialized = executeRequest(uri, getInitLibraryCallback()); + return library_initialized; } bool LibraryBridgeHelper::removeLibrary() { - startBridgeSync(); - auto uri = createRequestURI(LIB_DELETE_METHOD); - return executeRequest(uri); + /// Do not force bridge restart if it is not running in case of removeLibrary + /// because in this case after restart it will not have this dictionaty id in memory anyway. + if (bridgeHandShake()) + { + auto uri = createRequestURI(LIB_DELETE_METHOD); + return executeRequest(uri); + } + return true; } @@ -125,10 +208,12 @@ BlockInputStreamPtr LibraryBridgeHelper::loadAll() } -BlockInputStreamPtr LibraryBridgeHelper::loadIds(const std::string ids_string) +BlockInputStreamPtr LibraryBridgeHelper::loadIds(const std::vector & ids) { startBridgeSync(); auto uri = createRequestURI(LOAD_IDS_METHOD); + uri.addQueryParameter("ids_num", toString(ids.size())); /// Not used parameter, but helpful + auto ids_string = getDictIdsString(ids); return loadBase(uri, [ids_string](std::ostream & os) { os << ids_string; }); } @@ -149,13 +234,13 @@ BlockInputStreamPtr LibraryBridgeHelper::loadKeys(const Block & requested_block) } -bool LibraryBridgeHelper::executeRequest(const Poco::URI & uri, ReadWriteBufferFromHTTP::OutStreamCallback out_stream_callback) +bool LibraryBridgeHelper::executeRequest(const Poco::URI & uri, ReadWriteBufferFromHTTP::OutStreamCallback out_stream_callback) const { ReadWriteBufferFromHTTP buf( uri, Poco::Net::HTTPRequest::HTTP_POST, std::move(out_stream_callback), - ConnectionTimeouts::getHTTPTimeouts(getContext())); + http_timeouts); bool res; readBoolText(res, buf); @@ -169,7 +254,7 @@ BlockInputStreamPtr LibraryBridgeHelper::loadBase(const Poco::URI & uri, ReadWri uri, Poco::Net::HTTPRequest::HTTP_POST, std::move(out_stream_callback), - ConnectionTimeouts::getHTTPTimeouts(getContext()), + http_timeouts, 0, Poco::Net::HTTPBasicCredentials{}, DBMS_DEFAULT_BUFFER_SIZE, @@ -179,4 +264,13 @@ BlockInputStreamPtr LibraryBridgeHelper::loadBase(const Poco::URI & uri, ReadWri return std::make_shared>(input_stream, std::move(read_buf_ptr)); } + +String LibraryBridgeHelper::getDictIdsString(const std::vector & ids) +{ + WriteBufferFromOwnString out; + writeVectorBinary(ids, out); + return out.str(); +} + + } diff --git a/src/Bridge/LibraryBridgeHelper.h b/src/Bridge/LibraryBridgeHelper.h index 12fe0c33363..fec644240da 100644 --- a/src/Bridge/LibraryBridgeHelper.h +++ b/src/Bridge/LibraryBridgeHelper.h @@ -15,11 +15,18 @@ class LibraryBridgeHelper : public IBridgeHelper { public: + struct LibraryInitData + { + String library_path; + String library_settings; + String dict_attributes; + }; + static constexpr inline size_t DEFAULT_PORT = 9012; - LibraryBridgeHelper(ContextPtr context_, const Block & sample_block, const Field & dictionary_id_); + LibraryBridgeHelper(ContextPtr context_, const Block & sample_block, const Field & dictionary_id_, const LibraryInitData & library_data_); - bool initLibrary(const std::string & library_path, std::string library_settings, std::string attributes_names); + bool initLibrary(); bool cloneLibrary(const Field & other_dictionary_id); @@ -31,16 +38,19 @@ public: BlockInputStreamPtr loadAll(); - BlockInputStreamPtr loadIds(std::string ids_string); + BlockInputStreamPtr loadIds(const std::vector & ids); BlockInputStreamPtr loadKeys(const Block & requested_block); BlockInputStreamPtr loadBase(const Poco::URI & uri, ReadWriteBufferFromHTTP::OutStreamCallback out_stream_callback = {}); - bool executeRequest(const Poco::URI & uri, ReadWriteBufferFromHTTP::OutStreamCallback out_stream_callback = {}); + bool executeRequest(const Poco::URI & uri, ReadWriteBufferFromHTTP::OutStreamCallback out_stream_callback = {}) const; + LibraryInitData getLibraryData() const { return library_data; } protected: + bool bridgeHandShake() override; + void startBridge(std::unique_ptr cmd) const override; String serviceAlias() const override { return "clickhouse-library-bridge"; } @@ -61,6 +71,8 @@ protected: Poco::URI createBaseURI() const override; + ReadWriteBufferFromHTTP::OutStreamCallback getInitLibraryCallback() const; + private: static constexpr inline auto LIB_NEW_METHOD = "libNew"; static constexpr inline auto LIB_CLONE_METHOD = "libClone"; @@ -69,18 +81,24 @@ private: static constexpr inline auto LOAD_IDS_METHOD = "loadIds"; static constexpr inline auto LOAD_KEYS_METHOD = "loadKeys"; static constexpr inline auto IS_MODIFIED_METHOD = "isModified"; + static constexpr inline auto PING = "ping"; static constexpr inline auto SUPPORTS_SELECTIVE_LOAD_METHOD = "supportsSelectiveLoad"; Poco::URI createRequestURI(const String & method) const; + static String getDictIdsString(const std::vector & ids); + Poco::Logger * log; const Block sample_block; const Poco::Util::AbstractConfiguration & config; const Poco::Timespan http_timeout; + LibraryInitData library_data; Field dictionary_id; std::string bridge_host; size_t bridge_port; + bool library_initialized = false; + ConnectionTimeouts http_timeouts; }; } diff --git a/src/Bridge/XDBCBridgeHelper.h b/src/Bridge/XDBCBridgeHelper.h index 8be4c194962..b3613c381d0 100644 --- a/src/Bridge/XDBCBridgeHelper.h +++ b/src/Bridge/XDBCBridgeHelper.h @@ -60,20 +60,33 @@ public: static constexpr inline auto SCHEMA_ALLOWED_HANDLER = "/schema_allowed"; XDBCBridgeHelper( - ContextPtr context_, - Poco::Timespan http_timeout_, - const std::string & connection_string_) - : IXDBCBridgeHelper(context_->getGlobalContext()) - , log(&Poco::Logger::get(BridgeHelperMixin::getName() + "BridgeHelper")) - , connection_string(connection_string_) - , http_timeout(http_timeout_) - , config(context_->getGlobalContext()->getConfigRef()) -{ - bridge_host = config.getString(BridgeHelperMixin::configPrefix() + ".host", DEFAULT_HOST); - bridge_port = config.getUInt(BridgeHelperMixin::configPrefix() + ".port", DEFAULT_PORT); -} + ContextPtr context_, + Poco::Timespan http_timeout_, + const std::string & connection_string_) + : IXDBCBridgeHelper(context_->getGlobalContext()) + , log(&Poco::Logger::get(BridgeHelperMixin::getName() + "BridgeHelper")) + , connection_string(connection_string_) + , http_timeout(http_timeout_) + , config(context_->getGlobalContext()->getConfigRef()) + { + bridge_host = config.getString(BridgeHelperMixin::configPrefix() + ".host", DEFAULT_HOST); + bridge_port = config.getUInt(BridgeHelperMixin::configPrefix() + ".port", DEFAULT_PORT); + } protected: + bool bridgeHandShake() override + { + try + { + ReadWriteBufferFromHTTP buf(getPingURI(), Poco::Net::HTTPRequest::HTTP_GET, {}, ConnectionTimeouts::getHTTPTimeouts(getContext())); + return checkString(PING_OK_ANSWER, buf); + } + catch (...) + { + return false; + } + } + auto getConnectionString() const { return connection_string; } String getName() const override { return BridgeHelperMixin::getName(); } diff --git a/src/Core/Settings.h b/src/Core/Settings.h index 7e077d7eea6..d2f2a977fae 100644 --- a/src/Core/Settings.h +++ b/src/Core/Settings.h @@ -450,7 +450,6 @@ class IColumn; M(Bool, optimize_skip_merged_partitions, false, "Skip partitions with one part with level > 0 in optimize final", 0) \ M(Bool, optimize_on_insert, true, "Do the same transformation for inserted block of data as if merge was done on this block.", 0) \ M(Bool, allow_experimental_map_type, true, "Obsolete setting, does nothing.", 0) \ - M(Bool, allow_experimental_window_functions, false, "Allow experimental window functions", 0) \ M(Bool, allow_experimental_projection_optimization, false, "Enable projection optimization when processing SELECT queries", 0) \ M(Bool, force_optimize_projection, false, "If projection optimization is enabled, SELECT queries need to use projection", 0) \ M(Bool, async_socket_for_remote, true, "Asynchronously read from socket executing remote query", 0) \ diff --git a/src/DataStreams/tests/gtest_blocks_size_merging_streams.cpp b/src/DataStreams/tests/gtest_blocks_size_merging_streams.cpp index ed1ac6adbff..aa4c717a28b 100644 --- a/src/DataStreams/tests/gtest_blocks_size_merging_streams.cpp +++ b/src/DataStreams/tests/gtest_blocks_size_merging_streams.cpp @@ -83,7 +83,7 @@ TEST(MergingSortedTest, SimpleBlockSizeTest) EXPECT_EQ(pipe.numOutputPorts(), 3); auto transform = std::make_shared(pipe.getHeader(), pipe.numOutputPorts(), sort_description, - DEFAULT_MERGE_BLOCK_SIZE, 0, nullptr, false, true); + DEFAULT_MERGE_BLOCK_SIZE, 0, false, nullptr, false, true); pipe.addTransform(std::move(transform)); @@ -128,7 +128,7 @@ TEST(MergingSortedTest, MoreInterestingBlockSizes) EXPECT_EQ(pipe.numOutputPorts(), 3); auto transform = std::make_shared(pipe.getHeader(), pipe.numOutputPorts(), sort_description, - DEFAULT_MERGE_BLOCK_SIZE, 0, nullptr, false, true); + DEFAULT_MERGE_BLOCK_SIZE, 0, false, nullptr, false, true); pipe.addTransform(std::move(transform)); diff --git a/src/Databases/DatabaseReplicated.cpp b/src/Databases/DatabaseReplicated.cpp index 2c433c79033..ac6c525af68 100644 --- a/src/Databases/DatabaseReplicated.cpp +++ b/src/Databases/DatabaseReplicated.cpp @@ -40,6 +40,7 @@ namespace ErrorCodes extern const int NOT_IMPLEMENTED; extern const int INCORRECT_QUERY; extern const int ALL_CONNECTION_TRIES_FAILED; + extern const int NO_ACTIVE_REPLICAS; } static constexpr const char * DROPPED_MARK = "DROPPED"; @@ -137,7 +138,9 @@ ClusterPtr DatabaseReplicated::getClusterImpl() const Coordination::Stat stat; hosts = zookeeper->getChildren(zookeeper_path + "/replicas", &stat); if (hosts.empty()) - throw Exception(ErrorCodes::LOGICAL_ERROR, "No hosts found"); + throw Exception(ErrorCodes::NO_ACTIVE_REPLICAS, "No replicas of database {} found. " + "It's possible if the first replica is not fully created yet " + "or if the last replica was just dropped or due to logical error", database_name); Int32 cversion = stat.cversion; std::sort(hosts.begin(), hosts.end()); @@ -514,6 +517,19 @@ void DatabaseReplicated::recoverLostReplica(const ZooKeeperPtr & current_zookeep } } + auto make_query_context = [this, current_zookeeper]() + { + auto query_context = Context::createCopy(getContext()); + query_context->makeQueryContext(); + query_context->getClientInfo().query_kind = ClientInfo::QueryKind::SECONDARY_QUERY; + query_context->getClientInfo().is_replicated_database_internal = true; + query_context->setCurrentDatabase(database_name); + query_context->setCurrentQueryId(""); + auto txn = std::make_shared(current_zookeeper, zookeeper_path, false); + query_context->initZooKeeperMetadataTransaction(txn); + return query_context; + }; + String db_name = getDatabaseName(); String to_db_name = getDatabaseName() + BROKEN_TABLES_SUFFIX; if (total_tables * db_settings.max_broken_tables_ratio < tables_to_detach.size()) @@ -548,7 +564,7 @@ void DatabaseReplicated::recoverLostReplica(const ZooKeeperPtr & current_zookeep dropped_dictionaries += table->isDictionary(); table->flushAndShutdown(); - DatabaseAtomic::dropTable(getContext(), table_name, true); + DatabaseAtomic::dropTable(make_query_context(), table_name, true); } else { @@ -558,7 +574,7 @@ void DatabaseReplicated::recoverLostReplica(const ZooKeeperPtr & current_zookeep assert(db_name < to_db_name); DDLGuardPtr to_table_guard = DatabaseCatalog::instance().getDDLGuard(to_db_name, to_name); auto to_db_ptr = DatabaseCatalog::instance().getDatabase(to_db_name); - DatabaseAtomic::renameTable(getContext(), table_name, *to_db_ptr, to_name, false, false); + DatabaseAtomic::renameTable(make_query_context(), table_name, *to_db_ptr, to_name, false, false); ++moved_tables; } } @@ -577,7 +593,7 @@ void DatabaseReplicated::recoverLostReplica(const ZooKeeperPtr & current_zookeep /// TODO Maybe we should do it in two steps: rename all tables to temporary names and then rename them to actual names? DDLGuardPtr table_guard = DatabaseCatalog::instance().getDDLGuard(db_name, std::min(from, to)); DDLGuardPtr to_table_guard = DatabaseCatalog::instance().getDDLGuard(db_name, std::max(from, to)); - DatabaseAtomic::renameTable(getContext(), from, *this, to, false, false); + DatabaseAtomic::renameTable(make_query_context(), from, *this, to, false, false); } for (const auto & id : dropped_tables) @@ -592,15 +608,9 @@ void DatabaseReplicated::recoverLostReplica(const ZooKeeperPtr & current_zookeep } auto query_ast = parseQueryFromMetadataInZooKeeper(name_and_meta.first, name_and_meta.second); - - auto query_context = Context::createCopy(getContext()); - query_context->makeQueryContext(); - query_context->getClientInfo().query_kind = ClientInfo::QueryKind::SECONDARY_QUERY; - query_context->setCurrentDatabase(database_name); - query_context->setCurrentQueryId(""); // generate random query_id - LOG_INFO(log, "Executing {}", serializeAST(*query_ast)); - InterpreterCreateQuery(query_ast, query_context).execute(); + auto create_query_context = make_query_context(); + InterpreterCreateQuery(query_ast, create_query_context).execute(); } current_zookeeper->set(replica_path + "/log_ptr", toString(max_log_ptr)); diff --git a/src/Databases/DatabaseReplicatedWorker.cpp b/src/Databases/DatabaseReplicatedWorker.cpp index eb7e65e1b70..365a5d02816 100644 --- a/src/Databases/DatabaseReplicatedWorker.cpp +++ b/src/Databases/DatabaseReplicatedWorker.cpp @@ -60,12 +60,13 @@ void DatabaseReplicatedDDLWorker::initializeReplication() /// Check if we need to recover replica. /// Invariant: replica is lost if it's log_ptr value is less then max_log_ptr - logs_to_keep. - String log_ptr_str = current_zookeeper->get(database->replica_path + "/log_ptr"); + auto zookeeper = getAndSetZooKeeper(); + String log_ptr_str = zookeeper->get(database->replica_path + "/log_ptr"); UInt32 our_log_ptr = parse(log_ptr_str); - UInt32 max_log_ptr = parse(current_zookeeper->get(database->zookeeper_path + "/max_log_ptr")); - logs_to_keep = parse(current_zookeeper->get(database->zookeeper_path + "/logs_to_keep")); + UInt32 max_log_ptr = parse(zookeeper->get(database->zookeeper_path + "/max_log_ptr")); + logs_to_keep = parse(zookeeper->get(database->zookeeper_path + "/logs_to_keep")); if (our_log_ptr == 0 || our_log_ptr + logs_to_keep < max_log_ptr) - database->recoverLostReplica(current_zookeeper, our_log_ptr, max_log_ptr); + database->recoverLostReplica(zookeeper, our_log_ptr, max_log_ptr); else last_skipped_entry_name.emplace(DDLTaskBase::getLogEntryName(our_log_ptr)); } @@ -198,7 +199,7 @@ DDLTaskPtr DatabaseReplicatedDDLWorker::initAndCheckTask(const String & entry_na } } - UInt32 our_log_ptr = parse(current_zookeeper->get(fs::path(database->replica_path) / "log_ptr")); + UInt32 our_log_ptr = parse(zookeeper->get(fs::path(database->replica_path) / "log_ptr")); UInt32 entry_num = DatabaseReplicatedTask::getLogEntryNumber(entry_name); if (entry_num <= our_log_ptr) diff --git a/src/Dictionaries/LibraryDictionarySource.cpp b/src/Dictionaries/LibraryDictionarySource.cpp index 0b8b52a2d67..4747d3f1216 100644 --- a/src/Dictionaries/LibraryDictionarySource.cpp +++ b/src/Dictionaries/LibraryDictionarySource.cpp @@ -41,6 +41,9 @@ LibraryDictionarySource::LibraryDictionarySource( , sample_block{sample_block_} , context(Context::createCopy(context_)) { + if (fs::path(path).is_relative()) + path = fs::canonical(path); + if (created_from_ddl && !pathStartsWith(path, context->getDictionariesLibPath())) throw Exception(ErrorCodes::PATH_ACCESS_DENIED, "File path {} is not inside {}", path, context->getDictionariesLibPath()); @@ -48,17 +51,32 @@ LibraryDictionarySource::LibraryDictionarySource( throw Exception(ErrorCodes::FILE_DOESNT_EXIST, "LibraryDictionarySource: Can't load library {}: file doesn't exist", path); description.init(sample_block); - bridge_helper = std::make_shared(context, description.sample_block, dictionary_id); - auto res = bridge_helper->initLibrary(path, getLibrarySettingsString(config, config_prefix + ".settings"), getDictAttributesString()); - if (!res) + LibraryBridgeHelper::LibraryInitData library_data + { + .library_path = path, + .library_settings = getLibrarySettingsString(config, config_prefix + ".settings"), + .dict_attributes = getDictAttributesString() + }; + + bridge_helper = std::make_shared(context, description.sample_block, dictionary_id, library_data); + + if (!bridge_helper->initLibrary()) throw Exception(ErrorCodes::EXTERNAL_LIBRARY_ERROR, "Failed to create shared library from path: {}", path); } LibraryDictionarySource::~LibraryDictionarySource() { - bridge_helper->removeLibrary(); + try + { + bridge_helper->removeLibrary(); + } + catch (...) + { + tryLogCurrentException("LibraryDictionarySource"); + } + } @@ -72,8 +90,9 @@ LibraryDictionarySource::LibraryDictionarySource(const LibraryDictionarySource & , context(other.context) , description{other.description} { - bridge_helper = std::make_shared(context, description.sample_block, dictionary_id); - bridge_helper->cloneLibrary(other.dictionary_id); + bridge_helper = std::make_shared(context, description.sample_block, dictionary_id, other.bridge_helper->getLibraryData()); + if (!bridge_helper->cloneLibrary(other.dictionary_id)) + throw Exception(ErrorCodes::EXTERNAL_LIBRARY_ERROR, "Failed to clone library"); } @@ -99,7 +118,7 @@ BlockInputStreamPtr LibraryDictionarySource::loadAll() BlockInputStreamPtr LibraryDictionarySource::loadIds(const std::vector & ids) { LOG_TRACE(log, "loadIds {} size = {}", toString(), ids.size()); - return bridge_helper->loadIds(getDictIdsString(ids)); + return bridge_helper->loadIds(ids); } @@ -147,14 +166,6 @@ String LibraryDictionarySource::getLibrarySettingsString(const Poco::Util::Abstr } -String LibraryDictionarySource::getDictIdsString(const std::vector & ids) -{ - WriteBufferFromOwnString out; - writeVectorBinary(ids, out); - return out.str(); -} - - String LibraryDictionarySource::getDictAttributesString() { std::vector attributes_names(dict_struct.attributes.size()); diff --git a/src/Dictionaries/LibraryDictionarySource.h b/src/Dictionaries/LibraryDictionarySource.h index 88e133666e6..b5cf4f68b05 100644 --- a/src/Dictionaries/LibraryDictionarySource.h +++ b/src/Dictionaries/LibraryDictionarySource.h @@ -70,8 +70,6 @@ public: std::string toString() const override; private: - static String getDictIdsString(const std::vector & ids); - String getDictAttributesString(); static String getLibrarySettingsString(const Poco::Util::AbstractConfiguration & config, const std::string & config_root); @@ -82,7 +80,7 @@ private: const DictionaryStructure dict_struct; const std::string config_prefix; - const std::string path; + std::string path; const Field dictionary_id; Block sample_block; diff --git a/src/Disks/DiskLocal.cpp b/src/Disks/DiskLocal.cpp index 2897b08706d..b4d7f74f10f 100644 --- a/src/Disks/DiskLocal.cpp +++ b/src/Disks/DiskLocal.cpp @@ -31,6 +31,56 @@ std::mutex DiskLocal::reservation_mutex; using DiskLocalPtr = std::shared_ptr; +static void loadDiskLocalConfig(const String & name, + const Poco::Util::AbstractConfiguration & config, + const String & config_prefix, + ContextPtr context, + String & path, + UInt64 & keep_free_space_bytes) +{ + path = config.getString(config_prefix + ".path", ""); + if (name == "default") + { + if (!path.empty()) + throw Exception( + "\"default\" disk path should be provided in not it ", + ErrorCodes::UNKNOWN_ELEMENT_IN_CONFIG); + path = context->getPath(); + } + else + { + if (path.empty()) + throw Exception("Disk path can not be empty. Disk " + name, ErrorCodes::UNKNOWN_ELEMENT_IN_CONFIG); + if (path.back() != '/') + throw Exception("Disk path must end with /. Disk " + name, ErrorCodes::UNKNOWN_ELEMENT_IN_CONFIG); + } + + if (!FS::canRead(path) || !FS::canWrite(path)) + throw Exception("There is no RW access to the disk " + name + " (" + path + ")", ErrorCodes::PATH_ACCESS_DENIED); + + bool has_space_ratio = config.has(config_prefix + ".keep_free_space_ratio"); + + if (config.has(config_prefix + ".keep_free_space_bytes") && has_space_ratio) + throw Exception( + "Only one of 'keep_free_space_bytes' and 'keep_free_space_ratio' can be specified", + ErrorCodes::EXCESSIVE_ELEMENT_IN_CONFIG); + + keep_free_space_bytes = config.getUInt64(config_prefix + ".keep_free_space_bytes", 0); + + if (has_space_ratio) + { + auto ratio = config.getDouble(config_prefix + ".keep_free_space_ratio"); + if (ratio < 0 || ratio > 1) + throw Exception("'keep_free_space_ratio' have to be between 0 and 1", ErrorCodes::EXCESSIVE_ELEMENT_IN_CONFIG); + String tmp_path = path; + if (tmp_path.empty()) + tmp_path = context->getPath(); + + // Create tmp disk for getting total disk space. + keep_free_space_bytes = static_cast(DiskLocal("tmp", tmp_path, 0).getTotalSpace() * ratio); + } +} + class DiskLocalReservation : public IReservation { public: @@ -317,6 +367,21 @@ SyncGuardPtr DiskLocal::getDirectorySyncGuard(const String & path) const return std::make_unique(fs::path(disk_path) / path); } + +void DiskLocal::applyNewSettings(const Poco::Util::AbstractConfiguration & config, ContextPtr context, const String & config_prefix, const DisksMap &) +{ + String new_disk_path; + UInt64 new_keep_free_space_bytes; + + loadDiskLocalConfig(name, config, config_prefix, context, new_disk_path, new_keep_free_space_bytes); + + if (disk_path != new_disk_path) + throw Exception("Disk path can't be updated from config " + name, ErrorCodes::UNKNOWN_ELEMENT_IN_CONFIG); + + if (keep_free_space_bytes != new_keep_free_space_bytes) + keep_free_space_bytes = new_keep_free_space_bytes; +} + DiskPtr DiskLocalReservation::getDisk(size_t i) const { if (i != 0) @@ -334,7 +399,6 @@ void DiskLocalReservation::update(UInt64 new_size) disk->reserved_bytes += size; } - DiskLocalReservation::~DiskLocalReservation() { try @@ -369,48 +433,9 @@ void registerDiskLocal(DiskFactory & factory) const String & config_prefix, ContextPtr context, const DisksMap & /*map*/) -> DiskPtr { - String path = config.getString(config_prefix + ".path", ""); - if (name == "default") - { - if (!path.empty()) - throw Exception( - "\"default\" disk path should be provided in not it ", - ErrorCodes::UNKNOWN_ELEMENT_IN_CONFIG); - path = context->getPath(); - } - else - { - if (path.empty()) - throw Exception("Disk path can not be empty. Disk " + name, ErrorCodes::UNKNOWN_ELEMENT_IN_CONFIG); - if (path.back() != '/') - throw Exception("Disk path must end with /. Disk " + name, ErrorCodes::UNKNOWN_ELEMENT_IN_CONFIG); - } - - if (!FS::canRead(path) || !FS::canWrite(path)) - throw Exception("There is no RW access to the disk " + name + " (" + path + ")", ErrorCodes::PATH_ACCESS_DENIED); - - bool has_space_ratio = config.has(config_prefix + ".keep_free_space_ratio"); - - if (config.has(config_prefix + ".keep_free_space_bytes") && has_space_ratio) - throw Exception( - "Only one of 'keep_free_space_bytes' and 'keep_free_space_ratio' can be specified", - ErrorCodes::EXCESSIVE_ELEMENT_IN_CONFIG); - - UInt64 keep_free_space_bytes = config.getUInt64(config_prefix + ".keep_free_space_bytes", 0); - - if (has_space_ratio) - { - auto ratio = config.getDouble(config_prefix + ".keep_free_space_ratio"); - if (ratio < 0 || ratio > 1) - throw Exception("'keep_free_space_ratio' have to be between 0 and 1", ErrorCodes::EXCESSIVE_ELEMENT_IN_CONFIG); - String tmp_path = path; - if (tmp_path.empty()) - tmp_path = context->getPath(); - - // Create tmp disk for getting total disk space. - keep_free_space_bytes = static_cast(DiskLocal("tmp", tmp_path, 0).getTotalSpace() * ratio); - } - + String path; + UInt64 keep_free_space_bytes; + loadDiskLocalConfig(name, config, config_prefix, context, path, keep_free_space_bytes); return std::make_shared(name, path, keep_free_space_bytes); }; factory.registerDiskType("local", creator); diff --git a/src/Disks/DiskLocal.h b/src/Disks/DiskLocal.h index 3aa243b103b..0cf0cf39bbc 100644 --- a/src/Disks/DiskLocal.h +++ b/src/Disks/DiskLocal.h @@ -5,6 +5,7 @@ #include #include #include +#include namespace DB @@ -104,13 +105,15 @@ public: SyncGuardPtr getDirectorySyncGuard(const String & path) const override; + void applyNewSettings(const Poco::Util::AbstractConfiguration & config, ContextPtr context, const String & config_prefix, const DisksMap &) override; + private: bool tryReserve(UInt64 bytes); private: const String name; const String disk_path; - const UInt64 keep_free_space_bytes; + std::atomic keep_free_space_bytes; UInt64 reserved_bytes = 0; UInt64 reservation_count = 0; @@ -120,4 +123,5 @@ private: Poco::Logger * log = &Poco::Logger::get("DiskLocal"); }; + } diff --git a/src/Disks/DiskSelector.h b/src/Disks/DiskSelector.h index 88cc6ee5197..54752215441 100644 --- a/src/Disks/DiskSelector.h +++ b/src/Disks/DiskSelector.h @@ -32,7 +32,7 @@ public: /// Get all disks with names const DisksMap & getDisksMap() const { return disks; } - void addToDiskMap(String name, DiskPtr disk) + void addToDiskMap(const String & name, DiskPtr disk) { disks.emplace(name, disk); } diff --git a/src/Disks/IDisk.h b/src/Disks/IDisk.h index 717c72420cc..61c805961ae 100644 --- a/src/Disks/IDisk.h +++ b/src/Disks/IDisk.h @@ -13,9 +13,9 @@ #include #include #include +#include "Poco/Util/AbstractConfiguration.h" #include #include -#include "Poco/Util/AbstractConfiguration.h" namespace fs = std::filesystem; diff --git a/src/Disks/S3/DiskS3.cpp b/src/Disks/S3/DiskS3.cpp index 1f1c73c32c3..6dd29165566 100644 --- a/src/Disks/S3/DiskS3.cpp +++ b/src/Disks/S3/DiskS3.cpp @@ -363,7 +363,8 @@ int DiskS3::readSchemaVersion(const String & source_bucket, const String & sourc settings->client, source_bucket, source_path + SCHEMA_VERSION_OBJECT, - settings->s3_max_single_read_retries); + settings->s3_max_single_read_retries, + DBMS_DEFAULT_BUFFER_SIZE); readIntText(version, buffer); diff --git a/src/Functions/FunctionsComparison.h b/src/Functions/FunctionsComparison.h index 389b150e381..b9c7c211b74 100644 --- a/src/Functions/FunctionsComparison.h +++ b/src/Functions/FunctionsComparison.h @@ -1218,17 +1218,36 @@ public: { return res; } - else if ((isColumnedAsDecimal(left_type) || isColumnedAsDecimal(right_type)) - // Comparing Date and DateTime64 requires implicit conversion, - // otherwise Date is treated as number. - && !(date_and_datetime && (isDate(left_type) || isDate(right_type)))) + else if ((isColumnedAsDecimal(left_type) || isColumnedAsDecimal(right_type))) { - // compare - if (!allowDecimalComparison(left_type, right_type) && !date_and_datetime) - throw Exception("No operation " + getName() + " between " + left_type->getName() + " and " + right_type->getName(), - ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); + // Comparing Date and DateTime64 requires implicit conversion, + if (date_and_datetime && (isDate(left_type) || isDate(right_type))) + { + DataTypePtr common_type = getLeastSupertype({left_type, right_type}); + ColumnPtr c0_converted = castColumn(col_with_type_and_name_left, common_type); + ColumnPtr c1_converted = castColumn(col_with_type_and_name_right, common_type); + return executeDecimal({c0_converted, common_type, "left"}, {c1_converted, common_type, "right"}); + } + else + { + // compare + if (!allowDecimalComparison(left_type, right_type) && !date_and_datetime) + throw Exception( + "No operation " + getName() + " between " + left_type->getName() + " and " + right_type->getName(), + ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); + return executeDecimal(col_with_type_and_name_left, col_with_type_and_name_right); + } - return executeDecimal(col_with_type_and_name_left, col_with_type_and_name_right); + } + else if (date_and_datetime) + { + DataTypePtr common_type = getLeastSupertype({left_type, right_type}); + ColumnPtr c0_converted = castColumn(col_with_type_and_name_left, common_type); + ColumnPtr c1_converted = castColumn(col_with_type_and_name_right, common_type); + if (!((res = executeNumLeftType(c0_converted.get(), c1_converted.get())) + || (res = executeNumLeftType(c0_converted.get(), c1_converted.get())))) + throw Exception("Date related common types can only be UInt32 or UInt64", ErrorCodes::LOGICAL_ERROR); + return res; } else if (left_type->equals(*right_type)) { diff --git a/src/Functions/Regexps.h b/src/Functions/Regexps.h index 11f3e31e22e..5da1224ab8c 100644 --- a/src/Functions/Regexps.h +++ b/src/Functions/Regexps.h @@ -113,12 +113,34 @@ namespace MultiRegexps ScratchPtr scratch; }; + class RegexpsConstructor + { + public: + RegexpsConstructor() = default; + + void setConstructor(std::function constructor_) { constructor = std::move(constructor_); } + + Regexps * operator()() + { + std::unique_lock lock(mutex); + if (regexp) + return &*regexp; + regexp = constructor(); + return &*regexp; + } + + private: + std::function constructor; + std::optional regexp; + std::mutex mutex; + }; + struct Pool { /// Mutex for finding in map. std::mutex mutex; /// Patterns + possible edit_distance to database and scratch. - std::map, std::optional>, Regexps> storage; + std::map, std::optional>, RegexpsConstructor> storage; }; template @@ -250,15 +272,19 @@ namespace MultiRegexps /// If not found, compile and let other threads wait. if (known_regexps.storage.end() == it) + { it = known_regexps.storage - .emplace( - std::pair{str_patterns, edit_distance}, - constructRegexps(str_patterns, edit_distance)) + .emplace(std::piecewise_construct, std::make_tuple(std::move(str_patterns), edit_distance), std::make_tuple()) .first; - /// If found, unlock and return the database. - lock.unlock(); + it->second.setConstructor([&str_patterns = it->first.first, edit_distance]() + { + return constructRegexps(str_patterns, edit_distance); + }); + } - return &it->second; + /// Unlock before possible construction. + lock.unlock(); + return it->second(); } } diff --git a/src/IO/ReadBufferFromS3.h b/src/IO/ReadBufferFromS3.h index 46fb79a4a14..e24d571b557 100644 --- a/src/IO/ReadBufferFromS3.h +++ b/src/IO/ReadBufferFromS3.h @@ -43,7 +43,7 @@ public: const String & bucket_, const String & key_, UInt64 max_single_read_retries_, - size_t buffer_size_ = DBMS_DEFAULT_BUFFER_SIZE); + size_t buffer_size_); bool nextImpl() override; diff --git a/src/IO/ReadHelpers.h b/src/IO/ReadHelpers.h index d8e31c18617..f985070788c 100644 --- a/src/IO/ReadHelpers.h +++ b/src/IO/ReadHelpers.h @@ -403,7 +403,6 @@ bool tryReadIntText(T & x, ReadBuffer & buf) // -V1071 * Differs in following: * - for numbers starting with zero, parsed only zero; * - symbol '+' before number is not supported; - * - symbols :;<=>? are parsed as some numbers. */ template void readIntTextUnsafe(T & x, ReadBuffer & buf) @@ -437,15 +436,12 @@ void readIntTextUnsafe(T & x, ReadBuffer & buf) while (!buf.eof()) { - /// This check is suddenly faster than - /// unsigned char c = *buf.position() - '0'; - /// if (c < 10) - /// for unknown reason on Xeon E5645. + unsigned char value = *buf.position() - '0'; - if ((*buf.position() & 0xF0) == 0x30) /// It makes sense to have this condition inside loop. + if (value < 10) { res *= 10; - res += *buf.position() & 0x0F; + res += value; ++buf.position(); } else diff --git a/src/IO/ya.make b/src/IO/ya.make index 0023c6d7dfe..9e35a062a96 100644 --- a/src/IO/ya.make +++ b/src/IO/ya.make @@ -5,7 +5,7 @@ LIBRARY() ADDINCL( contrib/libs/zstd/include - contrib/restricted/fast_float + contrib/restricted/fast_float/include ) PEERDIR( diff --git a/src/IO/ya.make.in b/src/IO/ya.make.in index 4c28475e0e0..0b579b0df37 100644 --- a/src/IO/ya.make.in +++ b/src/IO/ya.make.in @@ -4,7 +4,7 @@ LIBRARY() ADDINCL( contrib/libs/zstd/include - contrib/restricted/fast_float + contrib/restricted/fast_float/include ) PEERDIR( diff --git a/src/Interpreters/AsynchronousMetrics.cpp b/src/Interpreters/AsynchronousMetrics.cpp index 6b2940154f8..d708ff4f9e0 100644 --- a/src/Interpreters/AsynchronousMetrics.cpp +++ b/src/Interpreters/AsynchronousMetrics.cpp @@ -77,6 +77,7 @@ AsynchronousMetrics::AsynchronousMetrics( , update_period(update_period_seconds) , servers_to_start_before_tables(servers_to_start_before_tables_) , servers(servers_) + , log(&Poco::Logger::get("AsynchronousMetrics")) { #if defined(OS_LINUX) openFileIfExists("/proc/meminfo", meminfo); @@ -174,26 +175,39 @@ AsynchronousMetrics::AsynchronousMetrics( edac.back().second = openFileIfExists(edac_uncorrectable_file); } - if (std::filesystem::exists("/sys/block")) - { - for (const auto & device_dir : std::filesystem::directory_iterator("/sys/block")) - { - String device_name = device_dir.path().filename(); - - /// We are not interested in loopback devices. - if (device_name.starts_with("loop")) - continue; - - std::unique_ptr file = openFileIfExists(device_dir.path() / "stat"); - if (!file) - continue; - - block_devs[device_name] = std::move(file); - } - } + openBlockDevices(); #endif } +#if defined(OS_LINUX) +void AsynchronousMetrics::openBlockDevices() +{ + LOG_TRACE(log, "Scanning /sys/block"); + + if (!std::filesystem::exists("/sys/block")) + return; + + block_devices_rescan_delay.restart(); + + block_devs.clear(); + + for (const auto & device_dir : std::filesystem::directory_iterator("/sys/block")) + { + String device_name = device_dir.path().filename(); + + /// We are not interested in loopback devices. + if (device_name.starts_with("loop")) + continue; + + std::unique_ptr file = openFileIfExists(device_dir.path() / "stat"); + if (!file) + continue; + + block_devs[device_name] = std::move(file); + } +} +#endif + void AsynchronousMetrics::start() { /// Update once right now, to make metrics available just after server start @@ -550,7 +564,7 @@ void AsynchronousMetrics::update(std::chrono::system_clock::time_point update_ti /// Log only if difference is high. This is for convenience. The threshold is arbitrary. if (difference >= 1048576 || difference <= -1048576) - LOG_TRACE(&Poco::Logger::get("AsynchronousMetrics"), + LOG_TRACE(log, "MemoryTracking: was {}, peak {}, will set to {} (RSS), difference: {}", ReadableSize(amount), ReadableSize(peak), @@ -877,6 +891,11 @@ void AsynchronousMetrics::update(std::chrono::system_clock::time_point update_ti } } + /// Update list of block devices periodically + /// (i.e. someone may add new disk to RAID array) + if (block_devices_rescan_delay.elapsedSeconds() >= 300) + openBlockDevices(); + for (auto & [name, device] : block_devs) { try @@ -930,6 +949,16 @@ void AsynchronousMetrics::update(std::chrono::system_clock::time_point update_ti } catch (...) { + /// Try to reopen block devices in case of error + /// (i.e. ENOENT means that some disk had been replaced, and it may apperas with a new name) + try + { + openBlockDevices(); + } + catch (...) + { + tryLogCurrentException(__PRETTY_FUNCTION__); + } tryLogCurrentException(__PRETTY_FUNCTION__); } } @@ -1303,9 +1332,9 @@ void AsynchronousMetrics::update(std::chrono::system_clock::time_point update_ti new_values["AsynchronousMetricsCalculationTimeSpent"] = watch.elapsedSeconds(); /// Log the new metrics. - if (auto log = getContext()->getAsynchronousMetricLog()) + if (auto asynchronous_metric_log = getContext()->getAsynchronousMetricLog()) { - log->addValues(new_values); + asynchronous_metric_log->addValues(new_values); } first_run = false; diff --git a/src/Interpreters/AsynchronousMetrics.h b/src/Interpreters/AsynchronousMetrics.h index 07e117c4dd9..c8677ac3ced 100644 --- a/src/Interpreters/AsynchronousMetrics.h +++ b/src/Interpreters/AsynchronousMetrics.h @@ -3,6 +3,7 @@ #include #include #include +#include #include #include @@ -15,6 +16,11 @@ #include +namespace Poco +{ +class Logger; +} + namespace DB { @@ -175,12 +181,17 @@ private: std::unordered_map network_interface_stats; + Stopwatch block_devices_rescan_delay; + + void openBlockDevices(); #endif std::unique_ptr thread; void run(); void update(std::chrono::system_clock::time_point update_time); + + Poco::Logger * log; }; } diff --git a/src/Interpreters/ClientInfo.h b/src/Interpreters/ClientInfo.h index d6158a2d7d5..7c169e6ebb5 100644 --- a/src/Interpreters/ClientInfo.h +++ b/src/Interpreters/ClientInfo.h @@ -100,6 +100,8 @@ public: UInt64 distributed_depth = 0; + bool is_replicated_database_internal = false; + bool empty() const { return query_kind == QueryKind::NO_QUERY; } /** Serialization and deserialization. diff --git a/src/Interpreters/DDLTask.cpp b/src/Interpreters/DDLTask.cpp index 4fb44738d8d..8bebc2fb442 100644 --- a/src/Interpreters/DDLTask.cpp +++ b/src/Interpreters/DDLTask.cpp @@ -359,6 +359,7 @@ ContextMutablePtr DatabaseReplicatedTask::makeQueryContext(ContextPtr from_conte { auto query_context = DDLTaskBase::makeQueryContext(from_context, zookeeper); query_context->getClientInfo().query_kind = ClientInfo::QueryKind::SECONDARY_QUERY; + query_context->getClientInfo().is_replicated_database_internal = true; query_context->setCurrentDatabase(database->getDatabaseName()); auto txn = std::make_shared(zookeeper, database->zookeeper_path, is_initial_query); diff --git a/src/Interpreters/DDLTask.h b/src/Interpreters/DDLTask.h index 703d691a358..41ab9ec8058 100644 --- a/src/Interpreters/DDLTask.h +++ b/src/Interpreters/DDLTask.h @@ -196,7 +196,7 @@ public: void commit(); - ~ZooKeeperMetadataTransaction() { assert(isExecuted() || std::uncaught_exceptions()); } + ~ZooKeeperMetadataTransaction() { assert(isExecuted() || std::uncaught_exceptions() || ops.empty()); } }; ClusterPtr tryGetReplicatedDatabaseCluster(const String & cluster_name); diff --git a/src/Interpreters/ExpressionAnalyzer.cpp b/src/Interpreters/ExpressionAnalyzer.cpp index 66c1cb9ad7b..6245b297b36 100644 --- a/src/Interpreters/ExpressionAnalyzer.cpp +++ b/src/Interpreters/ExpressionAnalyzer.cpp @@ -613,18 +613,6 @@ void makeWindowDescriptionFromAST(const Context & context, void ExpressionAnalyzer::makeWindowDescriptions(ActionsDAGPtr actions) { - // Convenient to check here because at least we have the Context. - if (!syntax->window_function_asts.empty() && - !getContext()->getSettingsRef().allow_experimental_window_functions) - { - throw Exception(ErrorCodes::NOT_IMPLEMENTED, - "The support for window functions is experimental and will change" - " in backwards-incompatible ways in the future releases. Set" - " allow_experimental_window_functions = 1 to enable it." - " While processing '{}'", - syntax->window_function_asts[0]->formatForErrorMessage()); - } - // Window definitions from the WINDOW clause const auto * select_query = query->as(); if (select_query && select_query->window()) diff --git a/src/Interpreters/GlobalSubqueriesVisitor.h b/src/Interpreters/GlobalSubqueriesVisitor.h index 71ff7fd2e0e..6a87527dc9c 100644 --- a/src/Interpreters/GlobalSubqueriesVisitor.h +++ b/src/Interpreters/GlobalSubqueriesVisitor.h @@ -63,7 +63,7 @@ public: return; bool is_table = false; - ASTPtr subquery_or_table_name = ast; /// ASTTableIdentifier | ASTSubquery | ASTTableExpression + ASTPtr subquery_or_table_name; /// ASTTableIdentifier | ASTSubquery | ASTTableExpression if (const auto * ast_table_expr = ast->as()) { @@ -76,7 +76,14 @@ public: } } else if (ast->as()) + { + subquery_or_table_name = ast; is_table = true; + } + else if (ast->as()) + { + subquery_or_table_name = ast; + } if (!subquery_or_table_name) throw Exception("Global subquery requires subquery or table name", ErrorCodes::WRONG_GLOBAL_SUBQUERY); diff --git a/src/Interpreters/IJoin.h b/src/Interpreters/IJoin.h index c2cf007d823..8fa85de4951 100644 --- a/src/Interpreters/IJoin.h +++ b/src/Interpreters/IJoin.h @@ -37,7 +37,7 @@ public: virtual size_t getTotalRowCount() const = 0; virtual size_t getTotalByteCount() const = 0; - virtual bool alwaysReturnsEmptySet() const { return false; } + virtual bool alwaysReturnsEmptySet() const = 0; /// StorageJoin/Dictionary is already filled. No need to call addJoinedBlock. /// Different query plan is used for such joins. diff --git a/src/Interpreters/InterpreterAlterQuery.cpp b/src/Interpreters/InterpreterAlterQuery.cpp index 6f0af049d05..76e7afb7009 100644 --- a/src/Interpreters/InterpreterAlterQuery.cpp +++ b/src/Interpreters/InterpreterAlterQuery.cpp @@ -54,7 +54,7 @@ BlockIO InterpreterAlterQuery::execute() DatabasePtr database = DatabaseCatalog::instance().getDatabase(table_id.database_name); if (typeid_cast(database.get()) - && getContext()->getClientInfo().query_kind != ClientInfo::QueryKind::SECONDARY_QUERY) + && !getContext()->getClientInfo().is_replicated_database_internal) { auto guard = DatabaseCatalog::instance().getDDLGuard(table_id.database_name, table_id.table_name); guard->releaseTableLock(); @@ -100,7 +100,8 @@ BlockIO InterpreterAlterQuery::execute() if (typeid_cast(database.get())) { int command_types_count = !mutation_commands.empty() + !partition_commands.empty() + !live_view_commands.empty() + !alter_commands.empty(); - if (1 < command_types_count) + bool mixed_settings_amd_metadata_alter = alter_commands.hasSettingsAlterCommand() && !alter_commands.isSettingsAlter(); + if (1 < command_types_count || mixed_settings_amd_metadata_alter) throw Exception(ErrorCodes::NOT_IMPLEMENTED, "For Replicated databases it's not allowed " "to execute ALTERs of different types in single query"); } diff --git a/src/Interpreters/InterpreterCreateQuery.cpp b/src/Interpreters/InterpreterCreateQuery.cpp index 1f7e239b504..b3cbfdcd035 100644 --- a/src/Interpreters/InterpreterCreateQuery.cpp +++ b/src/Interpreters/InterpreterCreateQuery.cpp @@ -856,7 +856,7 @@ BlockIO InterpreterCreateQuery::createTable(ASTCreateQuery & create) auto guard = DatabaseCatalog::instance().getDDLGuard(database_name, create.table); if (auto* ptr = typeid_cast(database.get()); - ptr && getContext()->getClientInfo().query_kind != ClientInfo::QueryKind::SECONDARY_QUERY) + ptr && !getContext()->getClientInfo().is_replicated_database_internal) { create.database = database_name; guard->releaseTableLock(); @@ -950,7 +950,7 @@ BlockIO InterpreterCreateQuery::createTable(ASTCreateQuery & create) auto guard = DatabaseCatalog::instance().getDDLGuard(create.database, create.table); if (auto * ptr = typeid_cast(database.get()); - ptr && getContext()->getClientInfo().query_kind != ClientInfo::QueryKind::SECONDARY_QUERY) + ptr && !getContext()->getClientInfo().is_replicated_database_internal) { assertOrSetUUID(create, database); guard->releaseTableLock(); diff --git a/src/Interpreters/InterpreterDropQuery.cpp b/src/Interpreters/InterpreterDropQuery.cpp index d5f6c3260c3..0e15c6be27c 100644 --- a/src/Interpreters/InterpreterDropQuery.cpp +++ b/src/Interpreters/InterpreterDropQuery.cpp @@ -133,7 +133,7 @@ BlockIO InterpreterDropQuery::executeToTableImpl(ASTDropQuery & query, DatabaseP /// Prevents recursive drop from drop database query. The original query must specify a table. bool is_drop_or_detach_database = query_ptr->as()->table.empty(); bool is_replicated_ddl_query = typeid_cast(database.get()) && - getContext()->getClientInfo().query_kind != ClientInfo::QueryKind::SECONDARY_QUERY && + !getContext()->getClientInfo().is_replicated_database_internal && !is_drop_or_detach_database; AccessFlags drop_storage; @@ -426,6 +426,7 @@ void InterpreterDropQuery::executeDropQuery(ASTDropQuery::Kind kind, ContextPtr if (auto txn = current_context->getZooKeeperMetadataTransaction()) { /// For Replicated database + drop_context->getClientInfo().is_replicated_database_internal = true; drop_context->setQueryContext(std::const_pointer_cast(current_context)); drop_context->initZooKeeperMetadataTransaction(txn, true); } diff --git a/src/Interpreters/InterpreterRenameQuery.cpp b/src/Interpreters/InterpreterRenameQuery.cpp index 515559ad903..373953e7530 100644 --- a/src/Interpreters/InterpreterRenameQuery.cpp +++ b/src/Interpreters/InterpreterRenameQuery.cpp @@ -81,7 +81,7 @@ BlockIO InterpreterRenameQuery::executeToTables(const ASTRenameQuery & rename, c DatabasePtr database = database_catalog.getDatabase(elem.from_database_name); if (typeid_cast(database.get()) - && getContext()->getClientInfo().query_kind != ClientInfo::QueryKind::SECONDARY_QUERY) + && !getContext()->getClientInfo().is_replicated_database_internal) { if (1 < descriptions.size()) throw Exception(ErrorCodes::NOT_IMPLEMENTED, "Database {} is Replicated, " diff --git a/src/Interpreters/InterpreterSelectQuery.cpp b/src/Interpreters/InterpreterSelectQuery.cpp index 2872637025c..af9280d1ff0 100644 --- a/src/Interpreters/InterpreterSelectQuery.cpp +++ b/src/Interpreters/InterpreterSelectQuery.cpp @@ -1928,11 +1928,13 @@ void InterpreterSelectQuery::executeFetchColumns(QueryProcessingStage::Enum proc } } + /// If we don't have filtration, we can pushdown limit to reading stage for optimizations. + UInt64 limit = (query.hasFiltration() || query.groupBy()) ? 0 : getLimitForSorting(query, context); if (query_info.projection) query_info.projection->input_order_info - = query_info.projection->order_optimizer->getInputOrder(query_info.projection->desc->metadata, context); + = query_info.projection->order_optimizer->getInputOrder(query_info.projection->desc->metadata, context, limit); else - query_info.input_order_info = query_info.order_optimizer->getInputOrder(metadata_snapshot, context); + query_info.input_order_info = query_info.order_optimizer->getInputOrder(metadata_snapshot, context, limit); } StreamLocalLimits limits; @@ -2292,8 +2294,14 @@ void InterpreterSelectQuery::executeOrderOptimized(QueryPlan & query_plan, Input { const Settings & settings = context->getSettingsRef(); + const auto & query = getSelectQuery(); auto finish_sorting_step = std::make_unique( - query_plan.getCurrentDataStream(), input_sorting_info->order_key_prefix_descr, output_order_descr, settings.max_block_size, limit); + query_plan.getCurrentDataStream(), + input_sorting_info->order_key_prefix_descr, + output_order_descr, + settings.max_block_size, + limit, + query.hasFiltration()); query_plan.addStep(std::move(finish_sorting_step)); } diff --git a/src/Interpreters/JoinToSubqueryTransformVisitor.cpp b/src/Interpreters/JoinToSubqueryTransformVisitor.cpp index 8595d3a4179..eabdeaefc04 100644 --- a/src/Interpreters/JoinToSubqueryTransformVisitor.cpp +++ b/src/Interpreters/JoinToSubqueryTransformVisitor.cpp @@ -551,8 +551,6 @@ std::vector normalizeColumnNamesExtractNeeded( else needed_columns[*table_pos].no_clashes.emplace(ident->shortName()); } - else if (!got_alias) - throw Exception("Unknown column name '" + ident->name() + "'", ErrorCodes::UNKNOWN_IDENTIFIER); } return needed_columns; diff --git a/src/Interpreters/MergeJoin.h b/src/Interpreters/MergeJoin.h index 11e5dc86dc2..844c730de4f 100644 --- a/src/Interpreters/MergeJoin.h +++ b/src/Interpreters/MergeJoin.h @@ -32,6 +32,8 @@ public: size_t getTotalRowCount() const override { return right_blocks.row_count; } size_t getTotalByteCount() const override { return right_blocks.bytes; } + /// Has to be called only after setTotals()/mergeRightBlocks() + bool alwaysReturnsEmptySet() const override { return (is_right || is_inner) && min_max_right_blocks.empty(); } BlockInputStreamPtr createStreamWithNonJoinedRows(const Block & result_sample_block, UInt64 max_block_size) const override; diff --git a/src/Interpreters/ProcessList.cpp b/src/Interpreters/ProcessList.cpp index 951ff6420c4..06320f00dfa 100644 --- a/src/Interpreters/ProcessList.cpp +++ b/src/Interpreters/ProcessList.cpp @@ -6,6 +6,7 @@ #include #include #include +#include #include #include #include @@ -297,7 +298,10 @@ QueryStatus::QueryStatus( { } -QueryStatus::~QueryStatus() = default; +QueryStatus::~QueryStatus() +{ + assert(executors.empty()); +} void QueryStatus::setQueryStreams(const BlockIO & io) { @@ -351,6 +355,11 @@ CancellationCode QueryStatus::cancelQuery(bool kill) BlockInputStreamPtr input_stream; BlockOutputStreamPtr output_stream; + SCOPE_EXIT({ + std::lock_guard lock(query_streams_mutex); + for (auto * e : executors) + e->cancel(); + }); if (tryGetQueryStreams(input_stream, output_stream)) { @@ -366,6 +375,20 @@ CancellationCode QueryStatus::cancelQuery(bool kill) return CancellationCode::CancelSent; } +void QueryStatus::addPipelineExecutor(PipelineExecutor * e) +{ + std::lock_guard lock(query_streams_mutex); + assert(std::find(executors.begin(), executors.end(), e) == executors.end()); + executors.push_back(e); +} + +void QueryStatus::removePipelineExecutor(PipelineExecutor * e) +{ + std::lock_guard lock(query_streams_mutex); + assert(std::find(executors.begin(), executors.end(), e) != executors.end()); + std::erase_if(executors, [e](PipelineExecutor * x) { return x == e; }); +} + void QueryStatus::setUserProcessList(ProcessListForUser * user_process_list_) { diff --git a/src/Interpreters/ProcessList.h b/src/Interpreters/ProcessList.h index fca28b031db..1adad84c040 100644 --- a/src/Interpreters/ProcessList.h +++ b/src/Interpreters/ProcessList.h @@ -22,6 +22,7 @@ #include #include #include +#include namespace CurrentMetrics @@ -34,6 +35,7 @@ namespace DB struct Settings; class IAST; +class PipelineExecutor; struct ProcessListForUser; class QueryStatus; @@ -109,6 +111,9 @@ protected: BlockInputStreamPtr query_stream_in; BlockOutputStreamPtr query_stream_out; + /// Array of PipelineExecutors to be cancelled when a cancelQuery is received + std::vector executors; + enum QueryStreamsStatus { NotInitialized, @@ -183,6 +188,12 @@ public: CancellationCode cancelQuery(bool kill); bool isKilled() const { return is_killed; } + + /// Adds a pipeline to the QueryStatus + void addPipelineExecutor(PipelineExecutor * e); + + /// Removes a pipeline to the QueryStatus + void removePipelineExecutor(PipelineExecutor * e); }; diff --git a/src/Interpreters/executeQuery.cpp b/src/Interpreters/executeQuery.cpp index da1c19260b6..051040e390b 100644 --- a/src/Interpreters/executeQuery.cpp +++ b/src/Interpreters/executeQuery.cpp @@ -262,7 +262,11 @@ static void onExceptionBeforeStart(const String & query_for_logging, ContextPtr elem.query = query_for_logging; elem.normalized_query_hash = normalizedQueryHash(query_for_logging); - // We don't calculate query_kind, databases, tables and columns when the query isn't able to start + // Try log query_kind if ast is valid + if (ast) + elem.query_kind = ast->getQueryKindString(); + + // We don't calculate databases, tables and columns when the query isn't able to start elem.exception_code = getCurrentExceptionCode(); elem.exception = getCurrentExceptionMessage(false); @@ -672,7 +676,7 @@ static std::tuple executeQueryImpl( } /// Common code for finish and exception callbacks - auto status_info_to_query_log = [](QueryLogElement &element, const QueryStatusInfo &info, const ASTPtr query_ast, ContextMutablePtr context_ptr) mutable + auto status_info_to_query_log = [](QueryLogElement & element, const QueryStatusInfo & info, const ASTPtr query_ast, const ContextPtr context_ptr) mutable { DB::UInt64 query_time = info.elapsed_seconds * 1000000; ProfileEvents::increment(ProfileEvents::QueryTimeMicroseconds, query_time); @@ -706,6 +710,17 @@ static std::tuple executeQueryImpl( element.query_columns.insert(access_info.columns.begin(), access_info.columns.end()); element.query_projections.insert(access_info.projections.begin(), access_info.projections.end()); element.query_views.insert(access_info.views.begin(), access_info.views.end()); + + const auto & factories_info = context_ptr->getQueryFactoriesInfo(); + element.used_aggregate_functions = factories_info.aggregate_functions; + element.used_aggregate_function_combinators = factories_info.aggregate_function_combinators; + element.used_database_engines = factories_info.database_engines; + element.used_data_type_families = factories_info.data_type_families; + element.used_dictionaries = factories_info.dictionaries; + element.used_formats = factories_info.formats; + element.used_functions = factories_info.functions; + element.used_storages = factories_info.storages; + element.used_table_functions = factories_info.table_functions; }; /// Also make possible for caller to log successful query finish and exception during execution. @@ -777,20 +792,6 @@ static std::tuple executeQueryImpl( ReadableSize(elem.read_bytes / elapsed_seconds)); } - elem.thread_ids = std::move(info.thread_ids); - elem.profile_counters = std::move(info.profile_counters); - - const auto & factories_info = context->getQueryFactoriesInfo(); - elem.used_aggregate_functions = factories_info.aggregate_functions; - elem.used_aggregate_function_combinators = factories_info.aggregate_function_combinators; - elem.used_database_engines = factories_info.database_engines; - elem.used_data_type_families = factories_info.data_type_families; - elem.used_dictionaries = factories_info.dictionaries; - elem.used_formats = factories_info.formats; - elem.used_functions = factories_info.functions; - elem.used_storages = factories_info.storages; - elem.used_table_functions = factories_info.table_functions; - if (log_queries && elem.type >= log_queries_min_type && Int64(elem.query_duration_ms) >= log_queries_min_query_duration_ms) { if (auto query_log = context->getQueryLog()) @@ -1020,22 +1021,31 @@ void executeQuery( const auto * ast_query_with_output = dynamic_cast(ast.get()); WriteBuffer * out_buf = &ostr; - std::optional out_file_buf; + std::unique_ptr compressed_buffer; if (ast_query_with_output && ast_query_with_output->out_file) { if (!allow_into_outfile) throw Exception("INTO OUTFILE is not allowed", ErrorCodes::INTO_OUTFILE_NOT_ALLOWED); const auto & out_file = ast_query_with_output->out_file->as().value.safeGet(); - out_file_buf.emplace(out_file, DBMS_DEFAULT_BUFFER_SIZE, O_WRONLY | O_EXCL | O_CREAT); - out_buf = &*out_file_buf; + compressed_buffer = wrapWriteBufferWithCompressionMethod( + std::make_unique(out_file, DBMS_DEFAULT_BUFFER_SIZE, O_WRONLY | O_EXCL | O_CREAT), + chooseCompressionMethod(out_file, ""), + /* compression level = */ 3 + ); } String format_name = ast_query_with_output && (ast_query_with_output->format != nullptr) ? getIdentifierName(ast_query_with_output->format) : context->getDefaultFormat(); - auto out = FormatFactory::instance().getOutputStreamParallelIfPossible(format_name, *out_buf, streams.in->getHeader(), context, {}, output_format_settings); + auto out = FormatFactory::instance().getOutputStreamParallelIfPossible( + format_name, + compressed_buffer ? *compressed_buffer : *out_buf, + streams.in->getHeader(), + context, + {}, + output_format_settings); /// Save previous progress callback if any. TODO Do it more conveniently. auto previous_progress_callback = context->getProgressCallback(); @@ -1059,15 +1069,18 @@ void executeQuery( const ASTQueryWithOutput * ast_query_with_output = dynamic_cast(ast.get()); WriteBuffer * out_buf = &ostr; - std::optional out_file_buf; + std::unique_ptr compressed_buffer; if (ast_query_with_output && ast_query_with_output->out_file) { if (!allow_into_outfile) throw Exception("INTO OUTFILE is not allowed", ErrorCodes::INTO_OUTFILE_NOT_ALLOWED); const auto & out_file = typeid_cast(*ast_query_with_output->out_file).value.safeGet(); - out_file_buf.emplace(out_file, DBMS_DEFAULT_BUFFER_SIZE, O_WRONLY | O_EXCL | O_CREAT); - out_buf = &*out_file_buf; + compressed_buffer = wrapWriteBufferWithCompressionMethod( + std::make_unique(out_file, DBMS_DEFAULT_BUFFER_SIZE, O_WRONLY | O_EXCL | O_CREAT), + chooseCompressionMethod(out_file, ""), + /* compression level = */ 3 + ); } String format_name = ast_query_with_output && (ast_query_with_output->format != nullptr) @@ -1081,7 +1094,14 @@ void executeQuery( return std::make_shared(header); }); - auto out = FormatFactory::instance().getOutputFormatParallelIfPossible(format_name, *out_buf, pipeline.getHeader(), context, {}, output_format_settings); + auto out = FormatFactory::instance().getOutputFormatParallelIfPossible( + format_name, + compressed_buffer ? *compressed_buffer : *out_buf, + pipeline.getHeader(), + context, + {}, + output_format_settings); + out->setAutoFlush(); /// Save previous progress callback if any. TODO Do it more conveniently. diff --git a/src/Parsers/ASTAlterQuery.h b/src/Parsers/ASTAlterQuery.h index 5fc146a3072..0fd6d2805ea 100644 --- a/src/Parsers/ASTAlterQuery.h +++ b/src/Parsers/ASTAlterQuery.h @@ -225,6 +225,8 @@ public: return removeOnCluster(clone(), new_database); } + const char * getQueryKindString() const override { return "Alter"; } + protected: void formatQueryImpl(const FormatSettings & settings, FormatState & state, FormatStateStacked frame) const override; diff --git a/src/Parsers/ASTCreateQuery.h b/src/Parsers/ASTCreateQuery.h index c7be67d9b78..e4f2b628886 100644 --- a/src/Parsers/ASTCreateQuery.h +++ b/src/Parsers/ASTCreateQuery.h @@ -102,6 +102,8 @@ public: bool isView() const { return is_ordinary_view || is_materialized_view || is_live_view; } + const char * getQueryKindString() const override { return "Create"; } + protected: void formatQueryImpl(const FormatSettings & settings, FormatState & state, FormatStateStacked frame) const override; }; diff --git a/src/Parsers/ASTDropQuery.h b/src/Parsers/ASTDropQuery.h index b612618aaec..6e5fd5854d8 100644 --- a/src/Parsers/ASTDropQuery.h +++ b/src/Parsers/ASTDropQuery.h @@ -45,6 +45,8 @@ public: return removeOnCluster(clone(), new_database); } + const char * getQueryKindString() const override { return "Drop"; } + protected: void formatQueryImpl(const FormatSettings & settings, FormatState &, FormatStateStacked) const override; }; diff --git a/src/Parsers/ASTGrantQuery.h b/src/Parsers/ASTGrantQuery.h index 184f2e143ee..b0fb64cb33e 100644 --- a/src/Parsers/ASTGrantQuery.h +++ b/src/Parsers/ASTGrantQuery.h @@ -34,5 +34,6 @@ public: void replaceEmptyDatabase(const String & current_database); void replaceCurrentUserTag(const String & current_user_name) const; ASTPtr getRewrittenASTWithoutOnCluster(const std::string &) const override { return removeOnCluster(clone()); } + const char * getQueryKindString() const override { return is_revoke ? "Revoke" : "Grant"; } }; } diff --git a/src/Parsers/ASTInsertQuery.h b/src/Parsers/ASTInsertQuery.h index 982f310fdb3..a454f46c3f1 100644 --- a/src/Parsers/ASTInsertQuery.h +++ b/src/Parsers/ASTInsertQuery.h @@ -47,6 +47,8 @@ public: return res; } + const char * getQueryKindString() const override { return "Insert"; } + protected: void formatImpl(const FormatSettings & settings, FormatState & state, FormatStateStacked frame) const override; }; diff --git a/src/Parsers/ASTRenameQuery.h b/src/Parsers/ASTRenameQuery.h index 611f81dc9e9..5739e7839d3 100644 --- a/src/Parsers/ASTRenameQuery.h +++ b/src/Parsers/ASTRenameQuery.h @@ -61,6 +61,8 @@ public: return query_ptr; } + const char * getQueryKindString() const override { return "Rename"; } + protected: void formatQueryImpl(const FormatSettings & settings, FormatState &, FormatStateStacked) const override { diff --git a/src/Parsers/ASTSelectQuery.h b/src/Parsers/ASTSelectQuery.h index db4d7e76320..12817199d13 100644 --- a/src/Parsers/ASTSelectQuery.h +++ b/src/Parsers/ASTSelectQuery.h @@ -69,6 +69,8 @@ public: const ASTPtr limitLength() const { return getExpression(Expression::LIMIT_LENGTH); } const ASTPtr settings() const { return getExpression(Expression::SETTINGS); } + bool hasFiltration() const { return where() || prewhere() || having(); } + /// Set/Reset/Remove expression. void setExpression(Expression expr, ASTPtr && ast); @@ -95,6 +97,8 @@ public: void setFinal(); + const char * getQueryKindString() const override { return "Select"; } + protected: void formatImpl(const FormatSettings & settings, FormatState & state, FormatStateStacked frame) const override; diff --git a/src/Parsers/ASTSelectWithUnionQuery.h b/src/Parsers/ASTSelectWithUnionQuery.h index ecf03bb6a05..0465bdac3a6 100644 --- a/src/Parsers/ASTSelectWithUnionQuery.h +++ b/src/Parsers/ASTSelectWithUnionQuery.h @@ -16,6 +16,8 @@ public: ASTPtr clone() const override; void formatQueryImpl(const FormatSettings & settings, FormatState & state, FormatStateStacked frame) const override; + const char * getQueryKindString() const override { return "Select"; } + enum class Mode { Unspecified, diff --git a/src/Parsers/ASTSystemQuery.h b/src/Parsers/ASTSystemQuery.h index cbe82cd936f..fa7b6ece59a 100644 --- a/src/Parsers/ASTSystemQuery.h +++ b/src/Parsers/ASTSystemQuery.h @@ -86,6 +86,8 @@ public: return removeOnCluster(clone(), new_database); } + const char * getQueryKindString() const override { return "System"; } + protected: void formatImpl(const FormatSettings & settings, FormatState & state, FormatStateStacked frame) const override; diff --git a/src/Parsers/IAST.h b/src/Parsers/IAST.h index 143094e1d7a..94a7b1a52ab 100644 --- a/src/Parsers/IAST.h +++ b/src/Parsers/IAST.h @@ -231,6 +231,9 @@ public: void cloneChildren(); + // Return query_kind string representation of this AST query. + virtual const char * getQueryKindString() const { return ""; } + public: /// For syntax highlighting. static const char * hilite_keyword; diff --git a/src/Processors/Executors/PipelineExecutor.cpp b/src/Processors/Executors/PipelineExecutor.cpp index 8a86ce89fd9..b91c1caa4a5 100644 --- a/src/Processors/Executors/PipelineExecutor.cpp +++ b/src/Processors/Executors/PipelineExecutor.cpp @@ -45,6 +45,8 @@ PipelineExecutor::PipelineExecutor(Processors & processors_, QueryStatus * elem) try { graph = std::make_unique(processors); + if (process_list_element) + process_list_element->addPipelineExecutor(this); } catch (Exception & exception) { @@ -59,6 +61,12 @@ PipelineExecutor::PipelineExecutor(Processors & processors_, QueryStatus * elem) } } +PipelineExecutor::~PipelineExecutor() +{ + if (process_list_element) + process_list_element->removePipelineExecutor(this); +} + void PipelineExecutor::addChildlessProcessorsToStack(Stack & stack) { UInt64 num_processors = processors.size(); diff --git a/src/Processors/Executors/PipelineExecutor.h b/src/Processors/Executors/PipelineExecutor.h index 213446ad43f..0652d81addc 100644 --- a/src/Processors/Executors/PipelineExecutor.h +++ b/src/Processors/Executors/PipelineExecutor.h @@ -31,6 +31,7 @@ public: /// /// Explicit graph representation is built in constructor. Throws if graph is not correct. explicit PipelineExecutor(Processors & processors_, QueryStatus * elem = nullptr); + ~PipelineExecutor(); /// Execute pipeline in multiple threads. Must be called once. /// In case of exception during execution throws any occurred. @@ -127,7 +128,7 @@ private: ProcessorsMap processors_map; /// Now it's used to check if query was killed. - QueryStatus * process_list_element = nullptr; + QueryStatus * const process_list_element = nullptr; /// Graph related methods. bool expandPipeline(Stack & stack, UInt64 pid); diff --git a/src/Processors/Merges/AggregatingSortedTransform.h b/src/Processors/Merges/AggregatingSortedTransform.h index a0425d4c376..e8bf90c2b31 100644 --- a/src/Processors/Merges/AggregatingSortedTransform.h +++ b/src/Processors/Merges/AggregatingSortedTransform.h @@ -16,7 +16,7 @@ public: const Block & header, size_t num_inputs, SortDescription description_, size_t max_block_size) : IMergingTransform( - num_inputs, header, header, true, + num_inputs, header, header, /*have_all_inputs_=*/ true, /*has_limit_below_one_block_=*/ false, header, num_inputs, std::move(description_), diff --git a/src/Processors/Merges/CollapsingSortedTransform.h b/src/Processors/Merges/CollapsingSortedTransform.h index 9e6bd306eee..87c466f31e8 100644 --- a/src/Processors/Merges/CollapsingSortedTransform.h +++ b/src/Processors/Merges/CollapsingSortedTransform.h @@ -20,7 +20,7 @@ public: WriteBuffer * out_row_sources_buf_ = nullptr, bool use_average_block_sizes = false) : IMergingTransform( - num_inputs, header, header, true, + num_inputs, header, header, /*have_all_inputs_=*/ true, /*has_limit_below_one_block_=*/ false, header, num_inputs, std::move(description_), diff --git a/src/Processors/Merges/FinishAggregatingInOrderTransform.h b/src/Processors/Merges/FinishAggregatingInOrderTransform.h index 4f9e53bd7d5..6d5e334311f 100644 --- a/src/Processors/Merges/FinishAggregatingInOrderTransform.h +++ b/src/Processors/Merges/FinishAggregatingInOrderTransform.h @@ -19,7 +19,7 @@ public: SortDescription description, size_t max_block_size) : IMergingTransform( - num_inputs, header, header, true, + num_inputs, header, header, /*have_all_inputs_=*/ true, /*has_limit_below_one_block_=*/ false, header, num_inputs, params, diff --git a/src/Processors/Merges/GraphiteRollupSortedTransform.h b/src/Processors/Merges/GraphiteRollupSortedTransform.h index 5104801aa0d..46272f00eed 100644 --- a/src/Processors/Merges/GraphiteRollupSortedTransform.h +++ b/src/Processors/Merges/GraphiteRollupSortedTransform.h @@ -15,7 +15,7 @@ public: SortDescription description_, size_t max_block_size, Graphite::Params params_, time_t time_of_merge_) : IMergingTransform( - num_inputs, header, header, true, + num_inputs, header, header, /*have_all_inputs_=*/ true, /*has_limit_below_one_block_=*/ false, header, num_inputs, std::move(description_), diff --git a/src/Processors/Merges/IMergingTransform.cpp b/src/Processors/Merges/IMergingTransform.cpp index eff786b150f..cba78390c97 100644 --- a/src/Processors/Merges/IMergingTransform.cpp +++ b/src/Processors/Merges/IMergingTransform.cpp @@ -14,9 +14,11 @@ IMergingTransformBase::IMergingTransformBase( size_t num_inputs, const Block & input_header, const Block & output_header, - bool have_all_inputs_) + bool have_all_inputs_, + bool has_limit_below_one_block_) : IProcessor(InputPorts(num_inputs, input_header), {output_header}) , have_all_inputs(have_all_inputs_) + , has_limit_below_one_block(has_limit_below_one_block_) { } @@ -64,10 +66,7 @@ IProcessor::Status IMergingTransformBase::prepareInitializeInputs() continue; if (input_states[i].is_initialized) - { - // input.setNotNeeded(); continue; - } input.setNeeded(); @@ -77,12 +76,17 @@ IProcessor::Status IMergingTransformBase::prepareInitializeInputs() continue; } - auto chunk = input.pull(); + /// setNotNeeded after reading first chunk, because in optimismtic case + /// (e.g. with optimized 'ORDER BY primary_key LIMIT n' and small 'n') + /// we won't have to read any chunks anymore; + auto chunk = input.pull(has_limit_below_one_block); if (!chunk.hasRows()) { - if (!input.isFinished()) + { + input.setNeeded(); all_inputs_has_data = false; + } continue; } diff --git a/src/Processors/Merges/IMergingTransform.h b/src/Processors/Merges/IMergingTransform.h index ce673131ab6..8b0a44ae025 100644 --- a/src/Processors/Merges/IMergingTransform.h +++ b/src/Processors/Merges/IMergingTransform.h @@ -16,7 +16,8 @@ public: size_t num_inputs, const Block & input_header, const Block & output_header, - bool have_all_inputs_); + bool have_all_inputs_, + bool has_limit_below_one_block_); OutputPort & getOutputPort() { return outputs.front(); } @@ -66,6 +67,7 @@ private: std::vector input_states; std::atomic have_all_inputs; bool is_initialized = false; + bool has_limit_below_one_block = false; IProcessor::Status prepareInitializeInputs(); }; @@ -81,8 +83,9 @@ public: const Block & input_header, const Block & output_header, bool have_all_inputs_, + bool has_limit_below_one_block_, Args && ... args) - : IMergingTransformBase(num_inputs, input_header, output_header, have_all_inputs_) + : IMergingTransformBase(num_inputs, input_header, output_header, have_all_inputs_, has_limit_below_one_block_) , algorithm(std::forward(args) ...) { } diff --git a/src/Processors/Merges/MergingSortedTransform.cpp b/src/Processors/Merges/MergingSortedTransform.cpp index ec1bdc59683..92fafa4242c 100644 --- a/src/Processors/Merges/MergingSortedTransform.cpp +++ b/src/Processors/Merges/MergingSortedTransform.cpp @@ -13,12 +13,13 @@ MergingSortedTransform::MergingSortedTransform( SortDescription description_, size_t max_block_size, UInt64 limit_, + bool has_limit_below_one_block_, WriteBuffer * out_row_sources_buf_, bool quiet_, bool use_average_block_sizes, bool have_all_inputs_) : IMergingTransform( - num_inputs, header, header, have_all_inputs_, + num_inputs, header, header, have_all_inputs_, has_limit_below_one_block_, header, num_inputs, std::move(description_), diff --git a/src/Processors/Merges/MergingSortedTransform.h b/src/Processors/Merges/MergingSortedTransform.h index 93bd36d8aec..1fa9b1275bd 100644 --- a/src/Processors/Merges/MergingSortedTransform.h +++ b/src/Processors/Merges/MergingSortedTransform.h @@ -17,6 +17,7 @@ public: SortDescription description, size_t max_block_size, UInt64 limit_ = 0, + bool has_limit_below_one_block_ = false, WriteBuffer * out_row_sources_buf_ = nullptr, bool quiet_ = false, bool use_average_block_sizes = false, diff --git a/src/Processors/Merges/ReplacingSortedTransform.h b/src/Processors/Merges/ReplacingSortedTransform.h index 757e19e2cbe..e760cdf0d2b 100644 --- a/src/Processors/Merges/ReplacingSortedTransform.h +++ b/src/Processors/Merges/ReplacingSortedTransform.h @@ -18,7 +18,7 @@ public: WriteBuffer * out_row_sources_buf_ = nullptr, bool use_average_block_sizes = false) : IMergingTransform( - num_inputs, header, header, true, + num_inputs, header, header, /*have_all_inputs_=*/ true, /*has_limit_below_one_block_=*/ false, header, num_inputs, std::move(description_), diff --git a/src/Processors/Merges/SummingSortedTransform.h b/src/Processors/Merges/SummingSortedTransform.h index 22361bb1a44..0287caed5aa 100644 --- a/src/Processors/Merges/SummingSortedTransform.h +++ b/src/Processors/Merges/SummingSortedTransform.h @@ -19,7 +19,7 @@ public: const Names & partition_key_columns, size_t max_block_size) : IMergingTransform( - num_inputs, header, header, true, + num_inputs, header, header, /*have_all_inputs_=*/ true, /*has_limit_below_one_block_=*/ false, header, num_inputs, std::move(description_), diff --git a/src/Processors/Merges/VersionedCollapsingTransform.h b/src/Processors/Merges/VersionedCollapsingTransform.h index f593734c603..f260e20f1da 100644 --- a/src/Processors/Merges/VersionedCollapsingTransform.h +++ b/src/Processors/Merges/VersionedCollapsingTransform.h @@ -19,7 +19,7 @@ public: WriteBuffer * out_row_sources_buf_ = nullptr, bool use_average_block_sizes = false) : IMergingTransform( - num_inputs, header, header, true, + num_inputs, header, header, /*have_all_inputs_=*/ true, /*has_limit_below_one_block_=*/ false, header, num_inputs, std::move(description_), diff --git a/src/Processors/QueryPlan/FinishSortingStep.cpp b/src/Processors/QueryPlan/FinishSortingStep.cpp index a2e056b3029..718eeb96cd8 100644 --- a/src/Processors/QueryPlan/FinishSortingStep.cpp +++ b/src/Processors/QueryPlan/FinishSortingStep.cpp @@ -31,12 +31,14 @@ FinishSortingStep::FinishSortingStep( SortDescription prefix_description_, SortDescription result_description_, size_t max_block_size_, - UInt64 limit_) + UInt64 limit_, + bool has_filtration_) : ITransformingStep(input_stream_, input_stream_.header, getTraits(limit_)) , prefix_description(std::move(prefix_description_)) , result_description(std::move(result_description_)) , max_block_size(max_block_size_) , limit(limit_) + , has_filtration(has_filtration_) { /// TODO: check input_stream is sorted by prefix_description. output_stream->sort_description = result_description; @@ -58,11 +60,14 @@ void FinishSortingStep::transformPipeline(QueryPipeline & pipeline, const BuildQ if (pipeline.getNumStreams() > 1) { UInt64 limit_for_merging = (need_finish_sorting ? 0 : limit); + bool has_limit_below_one_block = !has_filtration && limit_for_merging && limit_for_merging < max_block_size; auto transform = std::make_shared( pipeline.getHeader(), pipeline.getNumStreams(), prefix_description, - max_block_size, limit_for_merging); + max_block_size, + limit_for_merging, + has_limit_below_one_block); pipeline.addTransform(std::move(transform)); } diff --git a/src/Processors/QueryPlan/FinishSortingStep.h b/src/Processors/QueryPlan/FinishSortingStep.h index 9fe031e792d..5ea3a6d91b5 100644 --- a/src/Processors/QueryPlan/FinishSortingStep.h +++ b/src/Processors/QueryPlan/FinishSortingStep.h @@ -13,8 +13,9 @@ public: const DataStream & input_stream_, SortDescription prefix_description_, SortDescription result_description_, - size_t max_block_size, - UInt64 limit); + size_t max_block_size_, + UInt64 limit_, + bool has_filtration_); String getName() const override { return "FinishSorting"; } @@ -31,6 +32,7 @@ private: SortDescription result_description; size_t max_block_size; UInt64 limit; + bool has_filtration; }; } diff --git a/src/Processors/QueryPlan/ReadFromMergeTree.cpp b/src/Processors/QueryPlan/ReadFromMergeTree.cpp index d633c669e07..f8c12449c7e 100644 --- a/src/Processors/QueryPlan/ReadFromMergeTree.cpp +++ b/src/Processors/QueryPlan/ReadFromMergeTree.cpp @@ -13,7 +13,7 @@ #include #include #include -#include +#include #include #include #include @@ -179,26 +179,32 @@ template ProcessorPtr ReadFromMergeTree::createSource( const RangesInDataPart & part, const Names & required_columns, - bool use_uncompressed_cache) + bool use_uncompressed_cache, + bool has_limit_below_one_block) { return std::make_shared( data, metadata_snapshot, part.data_part, max_block_size, preferred_block_size_bytes, - preferred_max_column_in_block_size_bytes, required_columns, part.ranges, use_uncompressed_cache, - prewhere_info, actions_settings, true, reader_settings, virt_column_names, part.part_index_in_query); + preferred_max_column_in_block_size_bytes, required_columns, part.ranges, use_uncompressed_cache, prewhere_info, + actions_settings, true, reader_settings, virt_column_names, part.part_index_in_query, has_limit_below_one_block); } Pipe ReadFromMergeTree::readInOrder( RangesInDataParts parts_with_range, Names required_columns, ReadType read_type, - bool use_uncompressed_cache) + bool use_uncompressed_cache, + UInt64 limit) { Pipes pipes; + /// For reading in order it makes sense to read only + /// one range per task to reduce number of read rows. + bool has_limit_below_one_block = read_type != ReadType::Default && limit && limit < max_block_size; + for (const auto & part : parts_with_range) { auto source = read_type == ReadType::InReverseOrder - ? createSource(part, required_columns, use_uncompressed_cache) - : createSource(part, required_columns, use_uncompressed_cache); + ? createSource(part, required_columns, use_uncompressed_cache, has_limit_below_one_block) + : createSource(part, required_columns, use_uncompressed_cache, has_limit_below_one_block); pipes.emplace_back(std::move(source)); } @@ -224,7 +230,7 @@ Pipe ReadFromMergeTree::read( return readFromPool(parts_with_range, required_columns, max_streams, min_marks_for_concurrent_read, use_uncompressed_cache); - auto pipe = readInOrder(parts_with_range, required_columns, read_type, use_uncompressed_cache); + auto pipe = readInOrder(parts_with_range, required_columns, read_type, use_uncompressed_cache, 0); /// Use ConcatProcessor to concat sources together. /// It is needed to read in parts order (and so in PK order) if single thread is used. @@ -403,7 +409,6 @@ Pipe ReadFromMergeTree::spreadMarkRangesAmongStreamsWithOrder( { RangesInDataPart part = parts_with_ranges.back(); parts_with_ranges.pop_back(); - size_t & marks_in_part = info.sum_marks_in_parts.back(); /// We will not take too few rows from a part. @@ -418,8 +423,13 @@ Pipe ReadFromMergeTree::spreadMarkRangesAmongStreamsWithOrder( MarkRanges ranges_to_get_from_part; + /// We take full part if it contains enough marks or + /// if we know limit and part contains less than 'limit' rows. + bool take_full_part = marks_in_part <= need_marks + || (input_order_info->limit && input_order_info->limit < part.getRowsCount()); + /// We take the whole part if it is small enough. - if (marks_in_part <= need_marks) + if (take_full_part) { ranges_to_get_from_part = part.ranges; @@ -449,6 +459,7 @@ Pipe ReadFromMergeTree::spreadMarkRangesAmongStreamsWithOrder( } parts_with_ranges.emplace_back(part); } + ranges_to_get_from_part = split_ranges(ranges_to_get_from_part, input_order_info->direction); new_parts.emplace_back(part.data_part, part.part_index_in_query, std::move(ranges_to_get_from_part)); } @@ -457,8 +468,8 @@ Pipe ReadFromMergeTree::spreadMarkRangesAmongStreamsWithOrder( ? ReadFromMergeTree::ReadType::InOrder : ReadFromMergeTree::ReadType::InReverseOrder; - pipes.emplace_back(read(std::move(new_parts), column_names, read_type, - requested_num_streams, info.min_marks_for_concurrent_read, info.use_uncompressed_cache)); + pipes.emplace_back(readInOrder(std::move(new_parts), column_names, read_type, + info.use_uncompressed_cache, input_order_info->limit)); } if (need_preliminary_merge) @@ -486,7 +497,8 @@ Pipe ReadFromMergeTree::spreadMarkRangesAmongStreamsWithOrder( pipe.getHeader(), pipe.numOutputPorts(), sort_description, - max_block_size); + max_block_size, + 0, true); pipe.addTransform(std::move(transform)); } diff --git a/src/Processors/QueryPlan/ReadFromMergeTree.h b/src/Processors/QueryPlan/ReadFromMergeTree.h index b82e027420b..e83746c3ff0 100644 --- a/src/Processors/QueryPlan/ReadFromMergeTree.h +++ b/src/Processors/QueryPlan/ReadFromMergeTree.h @@ -116,10 +116,10 @@ private: Pipe read(RangesInDataParts parts_with_range, Names required_columns, ReadType read_type, size_t max_streams, size_t min_marks_for_concurrent_read, bool use_uncompressed_cache); Pipe readFromPool(RangesInDataParts parts_with_ranges, Names required_columns, size_t max_streams, size_t min_marks_for_concurrent_read, bool use_uncompressed_cache); - Pipe readInOrder(RangesInDataParts parts_with_range, Names required_columns, ReadType read_type, bool use_uncompressed_cache); + Pipe readInOrder(RangesInDataParts parts_with_range, Names required_columns, ReadType read_type, bool use_uncompressed_cache, UInt64 limit); template - ProcessorPtr createSource(const RangesInDataPart & part, const Names & required_columns, bool use_uncompressed_cache); + ProcessorPtr createSource(const RangesInDataPart & part, const Names & required_columns, bool use_uncompressed_cache, bool has_limit_below_one_block); Pipe spreadMarkRangesAmongStreams( RangesInDataParts && parts_with_ranges, diff --git a/src/Processors/Transforms/MergeSortingTransform.cpp b/src/Processors/Transforms/MergeSortingTransform.cpp index 1806693db3a..ca78a29071e 100644 --- a/src/Processors/Transforms/MergeSortingTransform.cpp +++ b/src/Processors/Transforms/MergeSortingTransform.cpp @@ -200,6 +200,7 @@ void MergeSortingTransform::consume(Chunk chunk) description, max_merged_block_size, limit, + false, nullptr, quiet, use_average_block_sizes, diff --git a/src/Server/GRPCServer.cpp b/src/Server/GRPCServer.cpp index ac59f82d419..b90b0c33f17 100644 --- a/src/Server/GRPCServer.cpp +++ b/src/Server/GRPCServer.cpp @@ -5,6 +5,7 @@ #include #include #include +#include #include #include #include @@ -480,6 +481,40 @@ namespace }; + /// A boolean state protected by mutex able to wait until other thread sets it to a specific value. + class BoolState + { + public: + explicit BoolState(bool initial_value) : value(initial_value) {} + + bool get() const + { + std::lock_guard lock{mutex}; + return value; + } + + void set(bool new_value) + { + std::lock_guard lock{mutex}; + if (value == new_value) + return; + value = new_value; + changed.notify_all(); + } + + void wait(bool wanted_value) const + { + std::unique_lock lock{mutex}; + changed.wait(lock, [this, wanted_value]() { return value == wanted_value; }); + } + + private: + bool value; + mutable std::mutex mutex; + mutable std::condition_variable changed; + }; + + /// Handles a connection after a responder is started (i.e. after getting a new call). class Call { @@ -564,18 +599,15 @@ namespace UInt64 waited_for_client_writing = 0; /// The following fields are accessed both from call_thread and queue_thread. - std::atomic reading_query_info = false; + BoolState reading_query_info{false}; std::atomic failed_to_read_query_info = false; GRPCQueryInfo next_query_info_while_reading; std::atomic want_to_cancel = false; std::atomic check_query_info_contains_cancel_only = false; - std::atomic sending_result = false; + BoolState sending_result{false}; std::atomic failed_to_send_result = false; ThreadFromGlobalPool call_thread; - std::condition_variable read_finished; - std::condition_variable write_finished; - std::mutex dummy_mutex; /// Doesn't protect anything. }; Call::Call(CallType call_type_, std::unique_ptr responder_, IServer & iserver_, Poco::Logger * log_) @@ -610,6 +642,7 @@ namespace { try { + setThreadName("GRPCServerCall"); receiveQuery(); executeQuery(); processInput(); @@ -1230,8 +1263,7 @@ namespace { auto start_reading = [&] { - assert(!reading_query_info); - reading_query_info = true; + reading_query_info.set(true); responder->read(next_query_info_while_reading, [this](bool ok) { /// Called on queue_thread. @@ -1256,18 +1288,16 @@ namespace /// on queue_thread. failed_to_read_query_info = true; } - reading_query_info = false; - read_finished.notify_one(); + reading_query_info.set(false); }); }; auto finish_reading = [&] { - if (reading_query_info) + if (reading_query_info.get()) { Stopwatch client_writing_watch; - std::unique_lock lock{dummy_mutex}; - read_finished.wait(lock, [this] { return !reading_query_info; }); + reading_query_info.wait(false); waited_for_client_writing += client_writing_watch.elapsedNanoseconds(); } throwIfFailedToReadQueryInfo(); @@ -1430,11 +1460,10 @@ namespace /// Wait for previous write to finish. /// (gRPC doesn't allow to start sending another result while the previous is still being sending.) - if (sending_result) + if (sending_result.get()) { Stopwatch client_reading_watch; - std::unique_lock lock{dummy_mutex}; - write_finished.wait(lock, [this] { return !sending_result; }); + sending_result.wait(false); waited_for_client_reading += client_reading_watch.elapsedNanoseconds(); } throwIfFailedToSendResult(); @@ -1445,14 +1474,13 @@ namespace if (write_buffer) write_buffer->finalize(); - sending_result = true; + sending_result.set(true); auto callback = [this](bool ok) { /// Called on queue_thread. if (!ok) failed_to_send_result = true; - sending_result = false; - write_finished.notify_one(); + sending_result.set(false); }; Stopwatch client_reading_final_watch; @@ -1472,8 +1500,7 @@ namespace if (send_final_message) { /// Wait until the result is actually sent. - std::unique_lock lock{dummy_mutex}; - write_finished.wait(lock, [this] { return !sending_result; }); + sending_result.wait(false); waited_for_client_reading += client_reading_final_watch.elapsedNanoseconds(); throwIfFailedToSendResult(); LOG_TRACE(log, "Final result has been sent to the client"); @@ -1584,7 +1611,7 @@ private: { /// Called on call_thread. That's why we can't destroy the `call` right now /// (thread can't join to itself). Thus here we only move the `call` from - /// `current_call` to `finished_calls` and run() will actually destroy the `call`. + /// `current_calls` to `finished_calls` and run() will actually destroy the `call`. std::lock_guard lock{mutex}; auto it = current_calls.find(call); finished_calls.push_back(std::move(it->second)); @@ -1593,6 +1620,7 @@ private: void run() { + setThreadName("GRPCServerQueue"); while (true) { { diff --git a/src/Storages/AlterCommands.cpp b/src/Storages/AlterCommands.cpp index 9e9510b51a4..1bb8e6bb9b0 100644 --- a/src/Storages/AlterCommands.cpp +++ b/src/Storages/AlterCommands.cpp @@ -1276,6 +1276,11 @@ void AlterCommands::validate(const StorageInMemoryMetadata & metadata, ContextPt validateColumnsDefaultsAndGetSampleBlock(default_expr_list, all_columns.getAll(), context); } +bool AlterCommands::hasSettingsAlterCommand() const +{ + return std::any_of(begin(), end(), [](const AlterCommand & c) { return c.isSettingsAlter(); }); +} + bool AlterCommands::isSettingsAlter() const { return std::all_of(begin(), end(), [](const AlterCommand & c) { return c.isSettingsAlter(); }); diff --git a/src/Storages/AlterCommands.h b/src/Storages/AlterCommands.h index 6987de68f9c..60f4ad7d552 100644 --- a/src/Storages/AlterCommands.h +++ b/src/Storages/AlterCommands.h @@ -195,9 +195,12 @@ public: void apply(StorageInMemoryMetadata & metadata, ContextPtr context) const; /// At least one command modify settings. + bool hasSettingsAlterCommand() const; + + /// All commands modify settings only. bool isSettingsAlter() const; - /// At least one command modify comments. + /// All commands modify comments only. bool isCommentAlter() const; /// Return mutation commands which some storages may execute as part of diff --git a/src/Storages/MergeTree/MergeTreeBaseSelectProcessor.cpp b/src/Storages/MergeTree/MergeTreeBaseSelectProcessor.cpp index 549a8a4f772..c91d60c5de7 100644 --- a/src/Storages/MergeTree/MergeTreeBaseSelectProcessor.cpp +++ b/src/Storages/MergeTree/MergeTreeBaseSelectProcessor.cpp @@ -465,6 +465,19 @@ Block MergeTreeBaseSelectProcessor::transformHeader( return block; } +std::unique_ptr MergeTreeBaseSelectProcessor::getSizePredictor( + const MergeTreeData::DataPartPtr & data_part, + const MergeTreeReadTaskColumns & task_columns, + const Block & sample_block) +{ + const auto & required_column_names = task_columns.columns.getNames(); + const auto & required_pre_column_names = task_columns.pre_columns.getNames(); + NameSet complete_column_names(required_column_names.begin(), required_column_names.end()); + complete_column_names.insert(required_pre_column_names.begin(), required_pre_column_names.end()); + + return std::make_unique( + data_part, Names(complete_column_names.begin(), complete_column_names.end()), sample_block); +} MergeTreeBaseSelectProcessor::~MergeTreeBaseSelectProcessor() = default; diff --git a/src/Storages/MergeTree/MergeTreeBaseSelectProcessor.h b/src/Storages/MergeTree/MergeTreeBaseSelectProcessor.h index 532dc48ec1e..d102e4f07a4 100644 --- a/src/Storages/MergeTree/MergeTreeBaseSelectProcessor.h +++ b/src/Storages/MergeTree/MergeTreeBaseSelectProcessor.h @@ -37,6 +37,11 @@ public: static Block transformHeader( Block block, const PrewhereInfoPtr & prewhere_info, const DataTypePtr & partition_value_type, const Names & virtual_columns); + static std::unique_ptr getSizePredictor( + const MergeTreeData::DataPartPtr & data_part, + const MergeTreeReadTaskColumns & task_columns, + const Block & sample_block); + protected: Chunk generate() final; diff --git a/src/Storages/MergeTree/MergeTreeBlockReadUtils.h b/src/Storages/MergeTree/MergeTreeBlockReadUtils.h index 31d609e4242..4c4081bd83b 100644 --- a/src/Storages/MergeTree/MergeTreeBlockReadUtils.h +++ b/src/Storages/MergeTree/MergeTreeBlockReadUtils.h @@ -70,7 +70,7 @@ struct MergeTreeReadTaskColumns /// column names to read during PREWHERE NamesAndTypesList pre_columns; /// resulting block may require reordering in accordance with `ordered_names` - bool should_reorder; + bool should_reorder = false; }; MergeTreeReadTaskColumns getReadTaskColumns( diff --git a/src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp b/src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp index a777c244426..6279d2d7d6f 100644 --- a/src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp +++ b/src/Storages/MergeTree/MergeTreeDataMergerMutator.cpp @@ -894,7 +894,7 @@ MergeTreeData::MutableDataPartPtr MergeTreeDataMergerMutator::mergePartsToTempor { case MergeTreeData::MergingParams::Ordinary: merged_transform = std::make_unique( - header, pipes.size(), sort_description, merge_block_size, 0, rows_sources_write_buf.get(), true, blocks_are_granules_size); + header, pipes.size(), sort_description, merge_block_size, 0, false, rows_sources_write_buf.get(), true, blocks_are_granules_size); break; case MergeTreeData::MergingParams::Collapsing: diff --git a/src/Storages/MergeTree/MergeTreeInOrderSelectProcessor.cpp b/src/Storages/MergeTree/MergeTreeInOrderSelectProcessor.cpp new file mode 100644 index 00000000000..48a9d62d872 --- /dev/null +++ b/src/Storages/MergeTree/MergeTreeInOrderSelectProcessor.cpp @@ -0,0 +1,54 @@ +#include + +namespace DB +{ + +namespace ErrorCodes +{ + extern const int MEMORY_LIMIT_EXCEEDED; +} + +bool MergeTreeInOrderSelectProcessor::getNewTask() +try +{ + if (all_mark_ranges.empty()) + { + finish(); + return false; + } + + if (!reader) + initializeReaders(); + + MarkRanges mark_ranges_for_task; + /// If we need to read few rows, set one range per task to reduce number of read data. + if (has_limit_below_one_block) + { + mark_ranges_for_task = { std::move(all_mark_ranges.front()) }; + all_mark_ranges.pop_front(); + } + else + { + mark_ranges_for_task = std::move(all_mark_ranges); + all_mark_ranges.clear(); + } + + auto size_predictor = (preferred_block_size_bytes == 0) ? nullptr + : getSizePredictor(data_part, task_columns, sample_block); + + task = std::make_unique( + data_part, mark_ranges_for_task, part_index_in_query, ordered_names, column_name_set, task_columns.columns, + task_columns.pre_columns, prewhere_info && prewhere_info->remove_prewhere_column, + task_columns.should_reorder, std::move(size_predictor)); + + return true; +} +catch (...) +{ + /// Suspicion of the broken part. A part is added to the queue for verification. + if (getCurrentExceptionCode() != ErrorCodes::MEMORY_LIMIT_EXCEEDED) + storage.reportBrokenPart(data_part->name); + throw; +} + +} diff --git a/src/Storages/MergeTree/MergeTreeInOrderSelectProcessor.h b/src/Storages/MergeTree/MergeTreeInOrderSelectProcessor.h new file mode 100644 index 00000000000..ecf648b0291 --- /dev/null +++ b/src/Storages/MergeTree/MergeTreeInOrderSelectProcessor.h @@ -0,0 +1,31 @@ +#pragma once +#include + +namespace DB +{ + + +/// Used to read data from single part with select query in order of primary key. +/// Cares about PREWHERE, virtual columns, indexes etc. +/// To read data from multiple parts, Storage (MergeTree) creates multiple such objects. +class MergeTreeInOrderSelectProcessor final : public MergeTreeSelectProcessor +{ +public: + template + MergeTreeInOrderSelectProcessor(Args &&... args) + : MergeTreeSelectProcessor{std::forward(args)...} + { + LOG_DEBUG(log, "Reading {} ranges in order from part {}, approx. {} rows starting from {}", + all_mark_ranges.size(), data_part->name, total_rows, + data_part->index_granularity.getMarkStartingRow(all_mark_ranges.front().begin)); + } + + String getName() const override { return "MergeTreeInOrder"; } + +private: + bool getNewTask() override; + + Poco::Logger * log = &Poco::Logger::get("MergeTreeInOrderSelectProcessor"); +}; + +} diff --git a/src/Storages/MergeTree/MergeTreeReadPool.cpp b/src/Storages/MergeTree/MergeTreeReadPool.cpp index f5ae5162676..b18f22c3ab1 100644 --- a/src/Storages/MergeTree/MergeTreeReadPool.cpp +++ b/src/Storages/MergeTree/MergeTreeReadPool.cpp @@ -228,29 +228,20 @@ std::vector MergeTreeReadPool::fillPerPartInfo( per_part_sum_marks.push_back(sum_marks); - auto [required_columns, required_pre_columns, should_reorder] = - getReadTaskColumns(data, metadata_snapshot, part.data_part, column_names, prewhere_info, check_columns); + auto task_columns = getReadTaskColumns(data, metadata_snapshot, part.data_part, column_names, prewhere_info, check_columns); - if (predict_block_size_bytes) - { - const auto & required_column_names = required_columns.getNames(); - const auto & required_pre_column_names = required_pre_columns.getNames(); - NameSet complete_column_names(required_column_names.begin(), required_column_names.end()); - complete_column_names.insert(required_pre_column_names.begin(), required_pre_column_names.end()); + auto size_predictor = !predict_block_size_bytes ? nullptr + : MergeTreeBaseSelectProcessor::getSizePredictor(part.data_part, task_columns, sample_block); - per_part_size_predictor.emplace_back(std::make_unique( - part.data_part, Names(complete_column_names.begin(), complete_column_names.end()), sample_block)); - } - else - per_part_size_predictor.emplace_back(nullptr); + per_part_size_predictor.emplace_back(std::move(size_predictor)); /// will be used to distinguish between PREWHERE and WHERE columns when applying filter - const auto & required_column_names = required_columns.getNames(); + const auto & required_column_names = task_columns.columns.getNames(); per_part_column_name_set.emplace_back(required_column_names.begin(), required_column_names.end()); - per_part_pre_columns.push_back(std::move(required_pre_columns)); - per_part_columns.push_back(std::move(required_columns)); - per_part_should_reorder.push_back(should_reorder); + per_part_pre_columns.push_back(std::move(task_columns.pre_columns)); + per_part_columns.push_back(std::move(task_columns.columns)); + per_part_should_reorder.push_back(task_columns.should_reorder); parts_with_idx.push_back({ part.data_part, part.part_index_in_query }); } diff --git a/src/Storages/MergeTree/MergeTreeReverseSelectProcessor.cpp b/src/Storages/MergeTree/MergeTreeReverseSelectProcessor.cpp index d546b2a95af..16ce9823ebb 100644 --- a/src/Storages/MergeTree/MergeTreeReverseSelectProcessor.cpp +++ b/src/Storages/MergeTree/MergeTreeReverseSelectProcessor.cpp @@ -1,8 +1,4 @@ #include -#include -#include -#include - namespace DB { @@ -12,74 +8,10 @@ namespace ErrorCodes extern const int MEMORY_LIMIT_EXCEEDED; } -MergeTreeReverseSelectProcessor::MergeTreeReverseSelectProcessor( - const MergeTreeData & storage_, - const StorageMetadataPtr & metadata_snapshot_, - const MergeTreeData::DataPartPtr & owned_data_part_, - UInt64 max_block_size_rows_, - size_t preferred_block_size_bytes_, - size_t preferred_max_column_in_block_size_bytes_, - Names required_columns_, - MarkRanges mark_ranges_, - bool use_uncompressed_cache_, - const PrewhereInfoPtr & prewhere_info_, - ExpressionActionsSettings actions_settings, - bool check_columns, - const MergeTreeReaderSettings & reader_settings_, - const Names & virt_column_names_, - size_t part_index_in_query_, - bool quiet) - : - MergeTreeBaseSelectProcessor{ - metadata_snapshot_->getSampleBlockForColumns(required_columns_, storage_.getVirtuals(), storage_.getStorageID()), - storage_, metadata_snapshot_, prewhere_info_, std::move(actions_settings), max_block_size_rows_, - preferred_block_size_bytes_, preferred_max_column_in_block_size_bytes_, - reader_settings_, use_uncompressed_cache_, virt_column_names_}, - required_columns{std::move(required_columns_)}, - data_part{owned_data_part_}, - all_mark_ranges(std::move(mark_ranges_)), - part_index_in_query(part_index_in_query_), - path(data_part->getFullRelativePath()) -{ - /// Let's estimate total number of rows for progress bar. - for (const auto & range : all_mark_ranges) - total_marks_count += range.end - range.begin; - - size_t total_rows = data_part->index_granularity.getRowsCountInRanges(all_mark_ranges); - - if (!quiet) - LOG_DEBUG(log, "Reading {} ranges in reverse order from part {}, approx. {} rows starting from {}", - all_mark_ranges.size(), data_part->name, total_rows, - data_part->index_granularity.getMarkStartingRow(all_mark_ranges.front().begin)); - - addTotalRowsApprox(total_rows); - - ordered_names = header_without_virtual_columns.getNames(); - - task_columns = getReadTaskColumns(storage, metadata_snapshot, data_part, required_columns, prewhere_info, check_columns); - - /// will be used to distinguish between PREWHERE and WHERE columns when applying filter - const auto & column_names = task_columns.columns.getNames(); - column_name_set = NameSet{column_names.begin(), column_names.end()}; - - if (use_uncompressed_cache) - owned_uncompressed_cache = storage.getContext()->getUncompressedCache(); - - owned_mark_cache = storage.getContext()->getMarkCache(); - - reader = data_part->getReader(task_columns.columns, metadata_snapshot, - all_mark_ranges, owned_uncompressed_cache.get(), - owned_mark_cache.get(), reader_settings); - - if (prewhere_info) - pre_reader = data_part->getReader(task_columns.pre_columns, metadata_snapshot, all_mark_ranges, - owned_uncompressed_cache.get(), owned_mark_cache.get(), reader_settings); -} - bool MergeTreeReverseSelectProcessor::getNewTask() try { - if ((chunks.empty() && all_mark_ranges.empty()) || total_marks_count == 0) + if (chunks.empty() && all_mark_ranges.empty()) { finish(); return false; @@ -90,21 +22,15 @@ try if (all_mark_ranges.empty()) return true; + if (!reader) + initializeReaders(); + /// Read ranges from right to left. MarkRanges mark_ranges_for_task = { all_mark_ranges.back() }; all_mark_ranges.pop_back(); - std::unique_ptr size_predictor; - if (preferred_block_size_bytes) - { - const auto & required_column_names = task_columns.columns.getNames(); - const auto & required_pre_column_names = task_columns.pre_columns.getNames(); - NameSet complete_column_names(required_column_names.begin(), required_column_names.end()); - complete_column_names.insert(required_pre_column_names.begin(), required_pre_column_names.end()); - - size_predictor = std::make_unique( - data_part, Names(complete_column_names.begin(), complete_column_names.end()), metadata_snapshot->getSampleBlock()); - } + auto size_predictor = (preferred_block_size_bytes == 0) ? nullptr + : getSizePredictor(data_part, task_columns, sample_block); task = std::make_unique( data_part, mark_ranges_for_task, part_index_in_query, ordered_names, column_name_set, @@ -150,17 +76,4 @@ Chunk MergeTreeReverseSelectProcessor::readFromPart() return res; } -void MergeTreeReverseSelectProcessor::finish() -{ - /** Close the files (before destroying the object). - * When many sources are created, but simultaneously reading only a few of them, - * buffers don't waste memory. - */ - reader.reset(); - pre_reader.reset(); - data_part.reset(); -} - -MergeTreeReverseSelectProcessor::~MergeTreeReverseSelectProcessor() = default; - } diff --git a/src/Storages/MergeTree/MergeTreeReverseSelectProcessor.h b/src/Storages/MergeTree/MergeTreeReverseSelectProcessor.h index a76564302a4..18ab51c03a0 100644 --- a/src/Storages/MergeTree/MergeTreeReverseSelectProcessor.h +++ b/src/Storages/MergeTree/MergeTreeReverseSelectProcessor.h @@ -1,77 +1,33 @@ #pragma once - -#include -#include -#include -#include -#include +#include namespace DB { - /// Used to read data from single part with select query +/// in reverse order of primary key. /// Cares about PREWHERE, virtual columns, indexes etc. /// To read data from multiple parts, Storage (MergeTree) creates multiple such objects. -class MergeTreeReverseSelectProcessor : public MergeTreeBaseSelectProcessor +class MergeTreeReverseSelectProcessor final : public MergeTreeSelectProcessor { public: - MergeTreeReverseSelectProcessor( - const MergeTreeData & storage, - const StorageMetadataPtr & metadata_snapshot, - const MergeTreeData::DataPartPtr & owned_data_part, - UInt64 max_block_size_rows, - size_t preferred_block_size_bytes, - size_t preferred_max_column_in_block_size_bytes, - Names required_columns_, - MarkRanges mark_ranges, - bool use_uncompressed_cache, - const PrewhereInfoPtr & prewhere_info, - ExpressionActionsSettings actions_settings, - bool check_columns, - const MergeTreeReaderSettings & reader_settings, - const Names & virt_column_names = {}, - size_t part_index_in_query = 0, - bool quiet = false); - - ~MergeTreeReverseSelectProcessor() override; + template + MergeTreeReverseSelectProcessor(Args &&... args) + : MergeTreeSelectProcessor{std::forward(args)...} + { + LOG_DEBUG(log, "Reading {} ranges in reverse order from part {}, approx. {} rows starting from {}", + all_mark_ranges.size(), data_part->name, total_rows, + data_part->index_granularity.getMarkStartingRow(all_mark_ranges.front().begin)); + } String getName() const override { return "MergeTreeReverse"; } - /// Closes readers and unlock part locks - void finish(); - -protected: - +private: bool getNewTask() override; Chunk readFromPart() override; -private: - Block header; - - /// Used by Task - Names required_columns; - /// Names from header. Used in order to order columns in read blocks. - Names ordered_names; - NameSet column_name_set; - - MergeTreeReadTaskColumns task_columns; - - /// Data part will not be removed if the pointer owns it - MergeTreeData::DataPartPtr data_part; - - /// Mark ranges we should read (in ascending order) - MarkRanges all_mark_ranges; - /// Total number of marks we should read - size_t total_marks_count = 0; - /// Value of _part_index virtual column (used only in SelectExecutor) - size_t part_index_in_query = 0; - - String path; - Chunks chunks; - Poco::Logger * log = &Poco::Logger::get("MergeTreeReverseSelectProcessor"); }; diff --git a/src/Storages/MergeTree/MergeTreeSelectProcessor.cpp b/src/Storages/MergeTree/MergeTreeSelectProcessor.cpp index 1e4b61e13d9..98077605f89 100644 --- a/src/Storages/MergeTree/MergeTreeSelectProcessor.cpp +++ b/src/Storages/MergeTree/MergeTreeSelectProcessor.cpp @@ -7,11 +7,6 @@ namespace DB { -namespace ErrorCodes -{ - extern const int MEMORY_LIMIT_EXCEEDED; -} - MergeTreeSelectProcessor::MergeTreeSelectProcessor( const MergeTreeData & storage_, const StorageMetadataPtr & metadata_snapshot_, @@ -28,96 +23,48 @@ MergeTreeSelectProcessor::MergeTreeSelectProcessor( const MergeTreeReaderSettings & reader_settings_, const Names & virt_column_names_, size_t part_index_in_query_, - bool quiet) - : - MergeTreeBaseSelectProcessor{ + bool has_limit_below_one_block_) + : MergeTreeBaseSelectProcessor{ metadata_snapshot_->getSampleBlockForColumns(required_columns_, storage_.getVirtuals(), storage_.getStorageID()), storage_, metadata_snapshot_, prewhere_info_, std::move(actions_settings), max_block_size_rows_, preferred_block_size_bytes_, preferred_max_column_in_block_size_bytes_, reader_settings_, use_uncompressed_cache_, virt_column_names_}, required_columns{std::move(required_columns_)}, data_part{owned_data_part_}, + sample_block(metadata_snapshot_->getSampleBlock()), all_mark_ranges(std::move(mark_ranges_)), part_index_in_query(part_index_in_query_), - check_columns(check_columns_) + has_limit_below_one_block(has_limit_below_one_block_), + check_columns(check_columns_), + total_rows(data_part->index_granularity.getRowsCountInRanges(all_mark_ranges)) { - /// Let's estimate total number of rows for progress bar. - for (const auto & range : all_mark_ranges) - total_marks_count += range.end - range.begin; - - size_t total_rows = data_part->index_granularity.getRowsCountInRanges(all_mark_ranges); - - if (!quiet) - LOG_DEBUG(log, "Reading {} ranges from part {}, approx. {} rows starting from {}", - all_mark_ranges.size(), data_part->name, total_rows, - data_part->index_granularity.getMarkStartingRow(all_mark_ranges.front().begin)); - addTotalRowsApprox(total_rows); ordered_names = header_without_virtual_columns.getNames(); } - -bool MergeTreeSelectProcessor::getNewTask() -try +void MergeTreeSelectProcessor::initializeReaders() { - /// Produce no more than one task - if (!is_first_task || total_marks_count == 0) - { - finish(); - return false; - } - is_first_task = false; - task_columns = getReadTaskColumns( storage, metadata_snapshot, data_part, required_columns, prewhere_info, check_columns); - std::unique_ptr size_predictor; - if (preferred_block_size_bytes) - { - const auto & required_column_names = task_columns.columns.getNames(); - const auto & required_pre_column_names = task_columns.pre_columns.getNames(); - NameSet complete_column_names(required_column_names.begin(), required_column_names.end()); - complete_column_names.insert(required_pre_column_names.begin(), required_pre_column_names.end()); - - size_predictor = std::make_unique( - data_part, Names(complete_column_names.begin(), complete_column_names.end()), metadata_snapshot->getSampleBlock()); - } - - /// will be used to distinguish between PREWHERE and WHERE columns when applying filter + /// Will be used to distinguish between PREWHERE and WHERE columns when applying filter const auto & column_names = task_columns.columns.getNames(); column_name_set = NameSet{column_names.begin(), column_names.end()}; - task = std::make_unique( - data_part, all_mark_ranges, part_index_in_query, ordered_names, column_name_set, task_columns.columns, - task_columns.pre_columns, prewhere_info && prewhere_info->remove_prewhere_column, - task_columns.should_reorder, std::move(size_predictor)); + if (use_uncompressed_cache) + owned_uncompressed_cache = storage.getContext()->getUncompressedCache(); - if (!reader) - { - if (use_uncompressed_cache) - owned_uncompressed_cache = storage.getContext()->getUncompressedCache(); + owned_mark_cache = storage.getContext()->getMarkCache(); - owned_mark_cache = storage.getContext()->getMarkCache(); + reader = data_part->getReader(task_columns.columns, metadata_snapshot, all_mark_ranges, + owned_uncompressed_cache.get(), owned_mark_cache.get(), reader_settings); - reader = data_part->getReader(task_columns.columns, metadata_snapshot, all_mark_ranges, + if (prewhere_info) + pre_reader = data_part->getReader(task_columns.pre_columns, metadata_snapshot, all_mark_ranges, owned_uncompressed_cache.get(), owned_mark_cache.get(), reader_settings); - if (prewhere_info) - pre_reader = data_part->getReader(task_columns.pre_columns, metadata_snapshot, all_mark_ranges, - owned_uncompressed_cache.get(), owned_mark_cache.get(), reader_settings); - } - - return true; } -catch (...) -{ - /// Suspicion of the broken part. A part is added to the queue for verification. - if (getCurrentExceptionCode() != ErrorCodes::MEMORY_LIMIT_EXCEEDED) - storage.reportBrokenPart(data_part->name); - throw; -} - void MergeTreeSelectProcessor::finish() { @@ -130,8 +77,6 @@ void MergeTreeSelectProcessor::finish() data_part.reset(); } - MergeTreeSelectProcessor::~MergeTreeSelectProcessor() = default; - } diff --git a/src/Storages/MergeTree/MergeTreeSelectProcessor.h b/src/Storages/MergeTree/MergeTreeSelectProcessor.h index e9712bd5ade..ea4cd349cba 100644 --- a/src/Storages/MergeTree/MergeTreeSelectProcessor.h +++ b/src/Storages/MergeTree/MergeTreeSelectProcessor.h @@ -1,5 +1,5 @@ #pragma once -#include +#include #include #include #include @@ -28,24 +28,21 @@ public: bool use_uncompressed_cache, const PrewhereInfoPtr & prewhere_info, ExpressionActionsSettings actions_settings, - bool check_columns, + bool check_columns_, const MergeTreeReaderSettings & reader_settings, const Names & virt_column_names = {}, - size_t part_index_in_query = 0, - bool quiet = false); + size_t part_index_in_query_ = 0, + bool has_limit_below_one_block_ = false); ~MergeTreeSelectProcessor() override; - String getName() const override { return "MergeTree"; } - /// Closes readers and unlock part locks void finish(); protected: - - bool getNewTask() override; - -private: + /// Defer initialization from constructor, because it may be heavy + /// and it's better to do it lazily in `getNewTask`, which is executing in parallel. + void initializeReaders(); /// Used by Task Names required_columns; @@ -58,17 +55,19 @@ private: /// Data part will not be removed if the pointer owns it MergeTreeData::DataPartPtr data_part; + /// Cache getSampleBlock call, which might be heavy. + Block sample_block; + /// Mark ranges we should read (in ascending order) MarkRanges all_mark_ranges; - /// Total number of marks we should read - size_t total_marks_count = 0; /// Value of _part_index virtual column (used only in SelectExecutor) size_t part_index_in_query = 0; + /// If true, every task will be created only with one range. + /// It reduces amount of read data for queries with small LIMIT. + bool has_limit_below_one_block = false; bool check_columns; - bool is_first_task = true; - - Poco::Logger * log = &Poco::Logger::get("MergeTreeSelectProcessor"); + size_t total_rows = 0; }; } diff --git a/src/Storages/ReadInOrderOptimizer.cpp b/src/Storages/ReadInOrderOptimizer.cpp index 87273330b34..912d284bfc0 100644 --- a/src/Storages/ReadInOrderOptimizer.cpp +++ b/src/Storages/ReadInOrderOptimizer.cpp @@ -37,7 +37,7 @@ ReadInOrderOptimizer::ReadInOrderOptimizer( array_join_result_to_source = syntax_result->array_join_result_to_source; } -InputOrderInfoPtr ReadInOrderOptimizer::getInputOrder(const StorageMetadataPtr & metadata_snapshot, ContextPtr context) const +InputOrderInfoPtr ReadInOrderOptimizer::getInputOrder(const StorageMetadataPtr & metadata_snapshot, ContextPtr context, UInt64 limit) const { Names sorting_key_columns = metadata_snapshot->getSortingKeyColumns(); if (!metadata_snapshot->hasSortingKey()) @@ -155,7 +155,8 @@ InputOrderInfoPtr ReadInOrderOptimizer::getInputOrder(const StorageMetadataPtr & if (order_key_prefix_descr.empty()) return {}; - return std::make_shared(std::move(order_key_prefix_descr), read_direction); + + return std::make_shared(std::move(order_key_prefix_descr), read_direction, limit); } } diff --git a/src/Storages/ReadInOrderOptimizer.h b/src/Storages/ReadInOrderOptimizer.h index 0abf2923a98..2686d081855 100644 --- a/src/Storages/ReadInOrderOptimizer.h +++ b/src/Storages/ReadInOrderOptimizer.h @@ -22,7 +22,7 @@ public: const SortDescription & required_sort_description, const TreeRewriterResultPtr & syntax_result); - InputOrderInfoPtr getInputOrder(const StorageMetadataPtr & metadata_snapshot, ContextPtr context) const; + InputOrderInfoPtr getInputOrder(const StorageMetadataPtr & metadata_snapshot, ContextPtr context, UInt64 limit = 0) const; private: /// Actions for every element of order expression to analyze functions for monotonicity diff --git a/src/Storages/SelectQueryInfo.h b/src/Storages/SelectQueryInfo.h index cf2c4d72f59..3b3c0fa1258 100644 --- a/src/Storages/SelectQueryInfo.h +++ b/src/Storages/SelectQueryInfo.h @@ -83,9 +83,10 @@ struct InputOrderInfo { SortDescription order_key_prefix_descr; int direction; + UInt64 limit; - InputOrderInfo(const SortDescription & order_key_prefix_descr_, int direction_) - : order_key_prefix_descr(order_key_prefix_descr_), direction(direction_) {} + InputOrderInfo(const SortDescription & order_key_prefix_descr_, int direction_, UInt64 limit_) + : order_key_prefix_descr(order_key_prefix_descr_), direction(direction_), limit(limit_) {} bool operator ==(const InputOrderInfo & other) const { diff --git a/src/Storages/StorageFactory.cpp b/src/Storages/StorageFactory.cpp index 5ca423b449a..cfa50b95487 100644 --- a/src/Storages/StorageFactory.cpp +++ b/src/Storages/StorageFactory.cpp @@ -222,7 +222,7 @@ StoragePtr StorageFactory::get( storage_def->engine->arguments->children = empty_engine_args; } - if (local_context->hasQueryContext() && context->getSettingsRef().log_queries) + if (local_context->hasQueryContext() && local_context->getSettingsRef().log_queries) local_context->getQueryContext()->addQueryFactoriesInfo(Context::QueryLogFactories::Storage, name); return res; diff --git a/src/Storages/StorageMongoDB.cpp b/src/Storages/StorageMongoDB.cpp index e27d16ecc68..1fd58a293dc 100644 --- a/src/Storages/StorageMongoDB.cpp +++ b/src/Storages/StorageMongoDB.cpp @@ -1,4 +1,5 @@ #include "StorageMongoDB.h" +#include "StorageMongoDBSocketFactory.h" #include #include @@ -33,6 +34,7 @@ StorageMongoDB::StorageMongoDB( const std::string & collection_name_, const std::string & username_, const std::string & password_, + const std::string & options_, const ColumnsDescription & columns_, const ConstraintsDescription & constraints_, const String & comment) @@ -43,6 +45,8 @@ StorageMongoDB::StorageMongoDB( , collection_name(collection_name_) , username(username_) , password(password_) + , options(options_) + , uri("mongodb://" + host_ + ":" + std::to_string(port_) + "/" + database_name_ + "?" + options_) { StorageInMemoryMetadata storage_metadata; storage_metadata.setColumns(columns_); @@ -56,7 +60,10 @@ void StorageMongoDB::connectIfNotConnected() { std::lock_guard lock{connection_mutex}; if (!connection) - connection = std::make_shared(host, port); + { + StorageMongoDBSocketFactory factory; + connection = std::make_shared(uri, factory); + } if (!authenticated) { @@ -102,9 +109,9 @@ void registerStorageMongoDB(StorageFactory & factory) { ASTs & engine_args = args.engine_args; - if (engine_args.size() != 5) + if (engine_args.size() < 5 || engine_args.size() > 6) throw Exception( - "Storage MongoDB requires 5 parameters: MongoDB('host:port', database, collection, 'user', 'password').", + "Storage MongoDB requires from 5 to 6 parameters: MongoDB('host:port', database, collection, 'user', 'password' [, 'options']).", ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH); for (auto & engine_arg : engine_args) @@ -118,6 +125,11 @@ void registerStorageMongoDB(StorageFactory & factory) const String & username = engine_args[3]->as().value.safeGet(); const String & password = engine_args[4]->as().value.safeGet(); + String options; + + if (engine_args.size() >= 6) + options = engine_args[5]->as().value.safeGet(); + return StorageMongoDB::create( args.table_id, parsed_host_port.first, @@ -126,6 +138,7 @@ void registerStorageMongoDB(StorageFactory & factory) collection, username, password, + options, args.columns, args.constraints, args.comment); diff --git a/src/Storages/StorageMongoDB.h b/src/Storages/StorageMongoDB.h index 2553acdd40c..3014b88a9ca 100644 --- a/src/Storages/StorageMongoDB.h +++ b/src/Storages/StorageMongoDB.h @@ -26,6 +26,7 @@ public: const std::string & collection_name_, const std::string & username_, const std::string & password_, + const std::string & options_, const ColumnsDescription & columns_, const ConstraintsDescription & constraints_, const String & comment); @@ -50,6 +51,8 @@ private: const std::string collection_name; const std::string username; const std::string password; + const std::string options; + const std::string uri; std::shared_ptr connection; bool authenticated = false; diff --git a/src/Storages/StorageMongoDBSocketFactory.cpp b/src/Storages/StorageMongoDBSocketFactory.cpp new file mode 100644 index 00000000000..9a2b120e9ed --- /dev/null +++ b/src/Storages/StorageMongoDBSocketFactory.cpp @@ -0,0 +1,55 @@ +#include "StorageMongoDBSocketFactory.h" + +#include + +#if !defined(ARCADIA_BUILD) +# include +#endif + +#include +#include + +#if USE_SSL +# include +#endif + + +namespace DB +{ + +namespace ErrorCodes +{ + extern const int FEATURE_IS_NOT_ENABLED_AT_BUILD_TIME; +} + +Poco::Net::StreamSocket StorageMongoDBSocketFactory::createSocket(const std::string & host, int port, Poco::Timespan connectTimeout, bool secure) +{ + return secure ? createSecureSocket(host, port, connectTimeout) : createPlainSocket(host, port, connectTimeout); +} + +Poco::Net::StreamSocket StorageMongoDBSocketFactory::createPlainSocket(const std::string & host, int port, Poco::Timespan connectTimeout) +{ + Poco::Net::SocketAddress address(host, port); + Poco::Net::StreamSocket socket; + + socket.connect(address, connectTimeout); + + return socket; +} + + +Poco::Net::StreamSocket StorageMongoDBSocketFactory::createSecureSocket(const std::string & host [[maybe_unused]], int port [[maybe_unused]], Poco::Timespan connectTimeout [[maybe_unused]]) +{ +#if USE_SSL + Poco::Net::SocketAddress address(host, port); + Poco::Net::SecureStreamSocket socket; + + socket.connect(address, connectTimeout); + + return socket; +#else + throw Exception("SSL is not enabled at build time.", ErrorCodes::FEATURE_IS_NOT_ENABLED_AT_BUILD_TIME); +#endif +} + +} diff --git a/src/Storages/StorageMongoDBSocketFactory.h b/src/Storages/StorageMongoDBSocketFactory.h new file mode 100644 index 00000000000..5fc423c63cb --- /dev/null +++ b/src/Storages/StorageMongoDBSocketFactory.h @@ -0,0 +1,19 @@ +#pragma once + +#include + + +namespace DB +{ + +class StorageMongoDBSocketFactory : public Poco::MongoDB::Connection::SocketFactory +{ +public: + virtual Poco::Net::StreamSocket createSocket(const std::string & host, int port, Poco::Timespan connectTimeout, bool secure) override; + +private: + static Poco::Net::StreamSocket createPlainSocket(const std::string & host, int port, Poco::Timespan connectTimeout); + static Poco::Net::StreamSocket createSecureSocket(const std::string & host, int port, Poco::Timespan connectTimeout); +}; + +} diff --git a/src/Storages/StorageReplicatedMergeTree.cpp b/src/Storages/StorageReplicatedMergeTree.cpp index 8b6267943bd..150a71a09e5 100644 --- a/src/Storages/StorageReplicatedMergeTree.cpp +++ b/src/Storages/StorageReplicatedMergeTree.cpp @@ -4972,6 +4972,8 @@ void StorageReplicatedMergeTree::alter( if (auto txn = query_context->getZooKeeperMetadataTransaction()) { + /// It would be better to clone ops instead of moving, so we could retry on ZBADVERSION, + /// but clone() is not implemented for Coordination::Request. txn->moveOpsTo(ops); /// NOTE: IDatabase::alterTable(...) is called when executing ALTER_METADATA queue entry without query context, /// so we have to update metadata of DatabaseReplicated here. @@ -5015,6 +5017,11 @@ void StorageReplicatedMergeTree::alter( throw Exception("Metadata on replica is not up to date with common metadata in Zookeeper. Cannot alter", ErrorCodes::CANNOT_ASSIGN_ALTER); + /// Cannot retry automatically, because some zookeeper ops were lost on the first attempt. Will retry on DDLWorker-level. + if (query_context->getZooKeeperMetadataTransaction()) + throw Exception("Cannot execute alter, because mutations version was suddenly changed due to concurrent alter", + ErrorCodes::CANNOT_ASSIGN_ALTER); + continue; } else @@ -5575,8 +5582,11 @@ void StorageReplicatedMergeTree::getStatus(Status & res, bool with_zk_fields) res.total_replicas = all_replicas.size(); for (const String & replica : all_replicas) - if (zookeeper->exists(fs::path(zookeeper_path) / "replicas" / replica / "is_active")) - ++res.active_replicas; + { + bool is_replica_active = zookeeper->exists(fs::path(zookeeper_path) / "replicas" / replica / "is_active"); + res.active_replicas += static_cast(is_replica_active); + res.replica_is_active.emplace(replica, is_replica_active); + } } catch (const Coordination::Exception &) { @@ -6007,6 +6017,10 @@ void StorageReplicatedMergeTree::mutate(const MutationCommands & commands, Conte } else if (rc == Coordination::Error::ZBADVERSION) { + /// Cannot retry automatically, because some zookeeper ops were lost on the first attempt. Will retry on DDLWorker-level. + if (query_context->getZooKeeperMetadataTransaction()) + throw Exception("Cannot execute alter, because mutations version was suddenly changed due to concurrent alter", + ErrorCodes::CANNOT_ASSIGN_ALTER); LOG_TRACE(log, "Version conflict when trying to create a mutation node, retrying..."); continue; } diff --git a/src/Storages/StorageReplicatedMergeTree.h b/src/Storages/StorageReplicatedMergeTree.h index 0f9d71bd5a5..3d2727d7bb9 100644 --- a/src/Storages/StorageReplicatedMergeTree.h +++ b/src/Storages/StorageReplicatedMergeTree.h @@ -176,6 +176,7 @@ public: UInt8 active_replicas; /// If the error has happened fetching the info from ZooKeeper, this field will be set. String zookeeper_exception; + std::unordered_map replica_is_active; }; /// Get the status of the table. If with_zk_fields = false - do not fill in the fields that require queries to ZK. diff --git a/src/Storages/StorageS3.cpp b/src/Storages/StorageS3.cpp index 1008bae346d..fc3ce3a10ed 100644 --- a/src/Storages/StorageS3.cpp +++ b/src/Storages/StorageS3.cpp @@ -208,7 +208,8 @@ bool StorageS3Source::initialize() file_path = fs::path(bucket) / current_key; read_buf = wrapReadBufferWithCompressionMethod( - std::make_unique(client, bucket, current_key, max_single_read_retries), chooseCompressionMethod(current_key, compression_hint)); + std::make_unique(client, bucket, current_key, max_single_read_retries, DBMS_DEFAULT_BUFFER_SIZE), + chooseCompressionMethod(current_key, compression_hint)); auto input_format = FormatFactory::instance().getInput(format, *read_buf, sample_block, getContext(), max_block_size); pipeline = std::make_unique(); pipeline->init(Pipe(input_format)); diff --git a/src/Storages/System/StorageSystemReplicas.cpp b/src/Storages/System/StorageSystemReplicas.cpp index fc33c6b421b..5c22d3c2fae 100644 --- a/src/Storages/System/StorageSystemReplicas.cpp +++ b/src/Storages/System/StorageSystemReplicas.cpp @@ -2,6 +2,7 @@ #include #include #include +#include #include #include #include @@ -51,6 +52,7 @@ StorageSystemReplicas::StorageSystemReplicas(const StorageID & table_id_) { "total_replicas", std::make_shared() }, { "active_replicas", std::make_shared() }, { "zookeeper_exception", std::make_shared() }, + { "replica_is_active", std::make_shared(std::make_shared(), std::make_shared()) } })); setInMemoryMetadata(storage_metadata); } @@ -101,7 +103,8 @@ Pipe StorageSystemReplicas::read( || column_name == "log_pointer" || column_name == "total_replicas" || column_name == "active_replicas" - || column_name == "zookeeper_exception") + || column_name == "zookeeper_exception" + || column_name == "replica_is_active") { with_zk_fields = true; break; @@ -184,6 +187,18 @@ Pipe StorageSystemReplicas::read( res_columns[col_num++]->insert(status.total_replicas); res_columns[col_num++]->insert(status.active_replicas); res_columns[col_num++]->insert(status.zookeeper_exception); + + Map replica_is_active_values; + for (const auto & [name, is_active] : status.replica_is_active) + { + Tuple is_replica_active_value; + is_replica_active_value.emplace_back(name); + is_replica_active_value.emplace_back(is_active); + + replica_is_active_values.emplace_back(std::move(is_replica_active_value)); + } + + res_columns[col_num++]->insert(std::move(replica_is_active_values)); } Block header = metadata_snapshot->getSampleBlock(); diff --git a/src/Storages/System/StorageSystemUsers.cpp b/src/Storages/System/StorageSystemUsers.cpp index 90b0a914d58..a48e12a1476 100644 --- a/src/Storages/System/StorageSystemUsers.cpp +++ b/src/Storages/System/StorageSystemUsers.cpp @@ -50,6 +50,7 @@ NamesAndTypesList StorageSystemUsers::getNamesAndTypes() {"grantees_any", std::make_shared()}, {"grantees_list", std::make_shared(std::make_shared())}, {"grantees_except", std::make_shared(std::make_shared())}, + {"default_database", std::make_shared()}, }; return names_and_types; } @@ -85,6 +86,7 @@ void StorageSystemUsers::fillData(MutableColumns & res_columns, ContextPtr conte auto & column_grantees_list_offsets = assert_cast(*res_columns[column_index++]).getOffsets(); auto & column_grantees_except = assert_cast(assert_cast(*res_columns[column_index]).getData()); auto & column_grantees_except_offsets = assert_cast(*res_columns[column_index++]).getOffsets(); + auto & column_default_database = assert_cast(*res_columns[column_index++]); auto add_row = [&](const String & name, const UUID & id, @@ -92,7 +94,8 @@ void StorageSystemUsers::fillData(MutableColumns & res_columns, ContextPtr conte const Authentication & authentication, const AllowedClientHosts & allowed_hosts, const RolesOrUsersSet & default_roles, - const RolesOrUsersSet & grantees) + const RolesOrUsersSet & grantees, + const String default_database) { column_name.insertData(name.data(), name.length()); column_id.push_back(id.toUnderType()); @@ -180,6 +183,8 @@ void StorageSystemUsers::fillData(MutableColumns & res_columns, ContextPtr conte for (const auto & except_name : grantees_ast->except_names) column_grantees_except.insertData(except_name.data(), except_name.length()); column_grantees_except_offsets.push_back(column_grantees_except.size()); + + column_default_database.insertData(default_database.data(),default_database.length()); }; for (const auto & id : ids) @@ -192,7 +197,8 @@ void StorageSystemUsers::fillData(MutableColumns & res_columns, ContextPtr conte if (!storage) continue; - add_row(user->getName(), id, storage->getStorageName(), user->authentication, user->allowed_client_hosts, user->default_roles, user->grantees); + add_row(user->getName(), id, storage->getStorageName(), user->authentication, user->allowed_client_hosts, + user->default_roles, user->grantees, user->default_database); } } diff --git a/src/Storages/ya.make b/src/Storages/ya.make index e794124362b..476449e8e6c 100644 --- a/src/Storages/ya.make +++ b/src/Storages/ya.make @@ -58,6 +58,7 @@ SRCS( MergeTree/MergeTreeDataSelectExecutor.cpp MergeTree/MergeTreeDataWriter.cpp MergeTree/MergeTreeDeduplicationLog.cpp + MergeTree/MergeTreeInOrderSelectProcessor.cpp MergeTree/MergeTreeIndexAggregatorBloomFilter.cpp MergeTree/MergeTreeIndexBloomFilter.cpp MergeTree/MergeTreeIndexConditionBloomFilter.cpp diff --git a/tests/clickhouse-test b/tests/clickhouse-test index 362d1d225d9..d83b3f08c42 100755 --- a/tests/clickhouse-test +++ b/tests/clickhouse-test @@ -27,6 +27,7 @@ except ImportError: import random import string import multiprocessing +import socket from contextlib import closing USE_JINJA = True @@ -267,6 +268,9 @@ def run_single_test(args, ext, server_logs_level, client_options, case_file, std os.system("LC_ALL=C sed -i -e 's|/auto_{{shard}}||g' {file}".format(file=stdout_file)) os.system("LC_ALL=C sed -i -e 's|auto_{{replica}}||g' {file}".format(file=stdout_file)) + # Normalize hostname in stdout file. + os.system("LC_ALL=C sed -i -e 's/{hostname}/localhost/g' {file}".format(hostname=socket.gethostname(), file=stdout_file)) + stdout = open(stdout_file, 'rb').read() if os.path.exists(stdout_file) else b'' stdout = str(stdout, errors='replace', encoding='utf-8') stderr = open(stderr_file, 'rb').read() if os.path.exists(stderr_file) else b'' diff --git a/tests/integration/helpers/cluster.py b/tests/integration/helpers/cluster.py index 720d5e2068b..6fe01b5df03 100644 --- a/tests/integration/helpers/cluster.py +++ b/tests/integration/helpers/cluster.py @@ -110,6 +110,7 @@ def subprocess_check_call(args, detach=False, nothrow=False): #logging.info('run:' + ' '.join(args)) return run_and_check(args, detach=detach, nothrow=nothrow) + def get_odbc_bridge_path(): path = os.environ.get('CLICKHOUSE_TESTS_ODBC_BRIDGE_BIN_PATH') if path is None: @@ -261,6 +262,7 @@ class ClickHouseCluster: self.with_hdfs = False self.with_kerberized_hdfs = False self.with_mongo = False + self.with_mongo_secure = False self.with_net_trics = False self.with_redis = False self.with_cassandra = False @@ -548,7 +550,6 @@ class ClickHouseCluster: return self.base_mysql_client_cmd - def setup_mysql_cmd(self, instance, env_variables, docker_compose_yml_dir): self.with_mysql = True env_variables['MYSQL_HOST'] = self.mysql_host @@ -680,6 +681,17 @@ class ClickHouseCluster: '--file', p.join(docker_compose_yml_dir, 'docker_compose_rabbitmq.yml')] return self.base_rabbitmq_cmd + def setup_mongo_secure_cmd(self, instance, env_variables, docker_compose_yml_dir): + self.with_mongo = self.with_mongo_secure = True + env_variables['MONGO_HOST'] = self.mongo_host + env_variables['MONGO_EXTERNAL_PORT'] = str(self.mongo_port) + env_variables['MONGO_INTERNAL_PORT'] = "27017" + env_variables['MONGO_CONFIG_PATH'] = HELPERS_DIR + self.base_cmd.extend(['--file', p.join(docker_compose_yml_dir, 'docker_compose_mongo_secure.yml')]) + self.base_mongo_cmd = ['docker-compose', '--env-file', instance.env_file, '--project-name', self.project_name, + '--file', p.join(docker_compose_yml_dir, 'docker_compose_mongo_secure.yml')] + return self.base_mongo_cmd + def setup_mongo_cmd(self, instance, env_variables, docker_compose_yml_dir): self.with_mongo = True env_variables['MONGO_HOST'] = self.mongo_host @@ -723,7 +735,8 @@ class ClickHouseCluster: macros=None, with_zookeeper=False, with_zookeeper_secure=False, with_mysql_client=False, with_mysql=False, with_mysql8=False, with_mysql_cluster=False, with_kafka=False, with_kerberized_kafka=False, with_rabbitmq=False, clickhouse_path_dir=None, - with_odbc_drivers=False, with_postgres=False, with_postgres_cluster=False, with_hdfs=False, with_kerberized_hdfs=False, with_mongo=False, + with_odbc_drivers=False, with_postgres=False, with_postgres_cluster=False, with_hdfs=False, + with_kerberized_hdfs=False, with_mongo=False, with_mongo_secure=False, with_redis=False, with_minio=False, with_cassandra=False, with_jdbc_bridge=False, hostname=None, env_variables=None, image="yandex/clickhouse-integration-test", tag=None, stay_alive=False, ipv4_address=None, ipv6_address=None, with_installed_binary=False, tmpfs=None, @@ -776,7 +789,7 @@ class ClickHouseCluster: with_kerberized_kafka=with_kerberized_kafka, with_rabbitmq=with_rabbitmq, with_kerberized_hdfs=with_kerberized_hdfs, - with_mongo=with_mongo, + with_mongo=with_mongo or with_mongo_secure, with_redis=with_redis, with_minio=with_minio, with_cassandra=with_cassandra, @@ -861,8 +874,11 @@ class ClickHouseCluster: if with_kerberized_hdfs and not self.with_kerberized_hdfs: cmds.append(self.setup_kerberized_hdfs_cmd(instance, env_variables, docker_compose_yml_dir)) - if with_mongo and not self.with_mongo: - cmds.append(self.setup_mongo_cmd(instance, env_variables, docker_compose_yml_dir)) + if (with_mongo or with_mongo_secure) and not (self.with_mongo or self.with_mongo_secure): + if with_mongo_secure: + cmds.append(self.setup_mongo_secure_cmd(instance, env_variables, docker_compose_yml_dir)) + else: + cmds.append(self.setup_mongo_cmd(instance, env_variables, docker_compose_yml_dir)) if self.with_net_trics: for cmd in cmds: @@ -1234,7 +1250,6 @@ class ClickHouseCluster: logging.debug("Waiting for Kafka to start up") time.sleep(1) - def wait_hdfs_to_start(self, timeout=300, check_marker=False): start = time.time() while time.time() - start < timeout: @@ -1251,9 +1266,11 @@ class ClickHouseCluster: raise Exception("Can't wait HDFS to start") - def wait_mongo_to_start(self, timeout=180): + def wait_mongo_to_start(self, timeout=30, secure=False): connection_str = 'mongodb://{user}:{password}@{host}:{port}'.format( host='localhost', port=self.mongo_port, user='root', password='clickhouse') + if secure: + connection_str += '/?tls=true&tlsAllowInvalidCertificates=true' connection = pymongo.MongoClient(connection_str) start = time.time() while time.time() - start < timeout: @@ -1320,7 +1337,6 @@ class ClickHouseCluster: raise Exception("Can't wait Schema Registry to start") - def wait_cassandra_to_start(self, timeout=180): self.cassandra_ip = self.get_instance_ip(self.cassandra_host) cass_client = cassandra.cluster.Cluster([self.cassandra_ip], port=self.cassandra_port, load_balancing_policy=RoundRobinPolicy()) @@ -1505,7 +1521,7 @@ class ClickHouseCluster: if self.with_mongo and self.base_mongo_cmd: logging.debug('Setup Mongo') run_and_check(self.base_mongo_cmd + common_opts) - self.wait_mongo_to_start(30) + self.wait_mongo_to_start(30, secure=self.with_mongo_secure) if self.with_redis and self.base_redis_cmd: logging.debug('Setup Redis') diff --git a/tests/integration/helpers/mongo_cert.pem b/tests/integration/helpers/mongo_cert.pem new file mode 100644 index 00000000000..9e18b1d4469 --- /dev/null +++ b/tests/integration/helpers/mongo_cert.pem @@ -0,0 +1,44 @@ +-----BEGIN RSA PRIVATE KEY----- +MIIEpAIBAAKCAQEAtz2fpa8hyUff8u8jYlh20HbkOO8hQi64Ke2Prack2Br0lhOr +1MI6I8nVk5iDrt+7ix2Cnt+2aZKb6HJv0CG1V25yWg+jgsXeIT1KHTJf8rTmYxhb +t+ye+S1Z0h/Rt+xqSd9XXfzOLPGHYfyx6ZQ4AumO/HoEFD4IH/qiREjwtOfRXuhz +CohqtUTyYR7pJmZqBSuGac461WVRisnjfKRxeVa3itc84/RgktgYej2x4PQBFk13 +xAXKrWmHkwdgWklTuuK8Gtoqz65Y4/J9CSl+Bd08QDdRnaVvq1u1eNTZg1BVyeRv +jFYBMSathKASrng5nK66Fdilw6tO/9khaP0SDQIDAQABAoIBAAm/5qGrKtIJ1/mW +Dbzq1g+Lc+MvngZmc/gPIsjrjsNM09y0WT0txGgpEgsTX1ZLoy/otw16+7qsSU1Z +4WcilAJ95umx0VJg8suz9iCNkJtaUrPNFPw5Q9AgQJo0hTUTCCi8EGr4y4OKqlhl +WJYEA+LryGbYmyT0k/wXmtClTOFjKS09mK4deQ1DqbBxayR9MUZgRJzEODA8eGXs +Rc6fJUenMVNMzIVLgpossRtKImoZNcf5UtCKL3HECunndQeMu4zuqLMU+EzL1F/o +iHKF7v3CVmsK0OxNJfOfT0abN3XaJttFwTJyghQjgP8OX1IKjlj3vo9xwEDfVUEf +GVIER0UCgYEA2j+kjaT3Dw2lNlBEsA8uvVlLJHBc9JBtMGduGYsIEgsL/iStqXv4 +xoA9N3CwkN/rrQpDfi/16DMXAAYdjGulPwiMNFBY22TCzhtPC2mAnfaSForxwZCs +lwc3KkIloo3N5XvN78AuZf8ewiS+bOEj+HHHqqSb1+u/csuaXO9neesCgYEA1u/I +Mlt/pxJkH+c3yOskwCh/CNhq9szi8G9LXROIQ58BT2ydJSEPpt7AhUTtQGimQQTW +KLiffJSkjuVaFckR1GjCoAmFGYw9wUb+TmFNScz5pJ2dXse8aBysAMIQfEIcRAEa +gKnkLBH6nw3+/Hm3xwoBc35t8Pa2ek7LsWDfbecCgYBhilQW4gVw+t49uf4Y2ZBA +G+pTbMx+mRXTrkYssFB5D+raOLZMqxVyUdoKLxkahpkkCxRDD1hN4JeE8Ta/jVSb +KUzQDKDJ3OybhOT86rgK4SpFXO/TXL9l+FmVT17WmZ3N1Fkjr7aM60pp5lYc/zo+ +TUu5XjwwcjJsMcbZhj2u5QKBgQCDNuUn4PYAP9kCJPzIWs0XxmEvPDeorZIJqFgA +3XC9n2+EVlFlHlbYz3oGofqY7Io6fUJkn7k1q+T+G4QwcozA+KeAXe90lkoJGVcc +8IfnewwYc+RjvVoG0SIsYE0CHrX0yhus2oqiYON4gGnfJkuMZk5WfKOPjH4AEuSF +SBd+lwKBgQCHG/DA6u2mYmezPF9hebWFoyAVSr2PDXDhu8cNNHCpx9GewJXhuK/P +tW8mazHzUuJKBvmaUXDIXFh4K6FNhjH16p5jR1w3hsPE7NEZhjfVRaUYPmBqaOYR +jp8H+Sh5g4Rwbtfp6Qhu6UAKi/y6Vozs5GkJtSiNrjNDVrD+sGGrXA== +-----END RSA PRIVATE KEY----- +-----BEGIN CERTIFICATE----- +MIICqDCCAZACFBdaMnuT0pWhmrh05UT3HXJ+kI0yMA0GCSqGSIb3DQEBCwUAMA0x +CzAJBgNVBAMMAmNhMB4XDTIxMDQwNjE3MDQxNVoXDTIyMDQwNjE3MDQxNVowFDES +MBAGA1UEAwwJbG9jYWxob3N0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC +AQEAtz2fpa8hyUff8u8jYlh20HbkOO8hQi64Ke2Prack2Br0lhOr1MI6I8nVk5iD +rt+7ix2Cnt+2aZKb6HJv0CG1V25yWg+jgsXeIT1KHTJf8rTmYxhbt+ye+S1Z0h/R +t+xqSd9XXfzOLPGHYfyx6ZQ4AumO/HoEFD4IH/qiREjwtOfRXuhzCohqtUTyYR7p +JmZqBSuGac461WVRisnjfKRxeVa3itc84/RgktgYej2x4PQBFk13xAXKrWmHkwdg +WklTuuK8Gtoqz65Y4/J9CSl+Bd08QDdRnaVvq1u1eNTZg1BVyeRvjFYBMSathKAS +rng5nK66Fdilw6tO/9khaP0SDQIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQAct2If +isMLHIqyL9GjY4b0xcxF4svFU/DUwNanStmoFMW1ifPf1cCqeMzyQOxBCDdMs0RT +hBbDYHW0BMXDqYIr3Ktbu38/3iVyr3pb56YOCKy8yHXpmKEaUBhCknSLcQyvNfeS +tM+DWsKFTZfyR5px+WwXbGKVMYwLaTON+/wcv1MeKMig3CxluaCpEJVYYwAiUc4K +sgvQNAunwGmPLPoXtUnpR2ZWiQA5R6yjS1oIe+8vpryFP6kjhWs0HR0jZEtLulV5 +WXUuxkqTXiBIvYpsmusoR44e9rptwLbV1wL/LUScRt9ttqFM3N5/Pof+2UwkSjGB +GAyPmw0Pkqtt+lva +-----END CERTIFICATE----- diff --git a/tests/integration/helpers/mongo_secure.conf b/tests/integration/helpers/mongo_secure.conf new file mode 100644 index 00000000000..1128b16b546 --- /dev/null +++ b/tests/integration/helpers/mongo_secure.conf @@ -0,0 +1,5 @@ +net: + ssl: + mode: requireSSL + PEMKeyFile: /mongo/mongo_cert.pem + allowConnectionsWithoutCertificates: true diff --git a/tests/integration/test_library_bridge/configs/config.d/config.xml b/tests/integration/test_library_bridge/configs/config.d/config.xml index 9bea75fbb6f..7811c1e26b7 100644 --- a/tests/integration/test_library_bridge/configs/config.d/config.xml +++ b/tests/integration/test_library_bridge/configs/config.d/config.xml @@ -8,5 +8,9 @@ 10 /var/log/clickhouse-server/stderr.log /var/log/clickhouse-server/stdout.log + + /var/log/clickhouse-server/clickhouse-library-bridge.log + /var/log/clickhouse-server/clickhouse-library-bridge.err.log + trace diff --git a/tests/integration/test_library_bridge/test.py b/tests/integration/test_library_bridge/test.py index ba44918bd60..607afb6db5f 100644 --- a/tests/integration/test_library_bridge/test.py +++ b/tests/integration/test_library_bridge/test.py @@ -2,14 +2,30 @@ import os import os.path as p import pytest import time +import logging from helpers.cluster import ClickHouseCluster, run_and_check cluster = ClickHouseCluster(__file__) instance = cluster.add_instance('instance', - dictionaries=['configs/dictionaries/dict1.xml'], - main_configs=['configs/config.d/config.xml']) + dictionaries=['configs/dictionaries/dict1.xml'], main_configs=['configs/config.d/config.xml'], stay_alive=True) + + +def create_dict_simple(): + instance.query('DROP DICTIONARY IF EXISTS lib_dict_c') + instance.query(''' + CREATE DICTIONARY lib_dict_c (key UInt64, value1 UInt64, value2 UInt64, value3 UInt64) + PRIMARY KEY key SOURCE(library(PATH '/etc/clickhouse-server/config.d/dictionaries_lib/dict_lib.so')) + LAYOUT(CACHE( + SIZE_IN_CELLS 10000000 + BLOCK_SIZE 4096 + FILE_SIZE 16777216 + READ_BUFFER_SIZE 1048576 + MAX_STORED_KEYS 1048576)) + LIFETIME(2) ; + ''') + @pytest.fixture(scope="module") def ch_cluster(): @@ -98,6 +114,10 @@ def test_load_ids(ch_cluster): result = instance.query('''select dictGet(lib_dict_c, 'value1', toUInt64(0));''') assert(result.strip() == '100') + + # Just check bridge is ok with a large vector of random ids + instance.query('''select number, dictGet(lib_dict_c, 'value1', toUInt64(rand())) from numbers(1000);''') + result = instance.query('''select dictGet(lib_dict_c, 'value1', toUInt64(1));''') assert(result.strip() == '101') instance.query('DROP DICTIONARY lib_dict_c') @@ -160,6 +180,91 @@ def test_null_values(ch_cluster): assert(result == expected) +def test_recover_after_bridge_crash(ch_cluster): + if instance.is_built_with_memory_sanitizer(): + pytest.skip("Memory Sanitizer cannot work with third-party shared libraries") + + create_dict_simple() + + result = instance.query('''select dictGet(lib_dict_c, 'value1', toUInt64(0));''') + assert(result.strip() == '100') + result = instance.query('''select dictGet(lib_dict_c, 'value1', toUInt64(1));''') + assert(result.strip() == '101') + + instance.exec_in_container(['bash', '-c', 'kill -9 `pidof clickhouse-library-bridge`'], user='root') + instance.query('SYSTEM RELOAD DICTIONARY lib_dict_c') + + result = instance.query('''select dictGet(lib_dict_c, 'value1', toUInt64(0));''') + assert(result.strip() == '100') + result = instance.query('''select dictGet(lib_dict_c, 'value1', toUInt64(1));''') + assert(result.strip() == '101') + + instance.exec_in_container(['bash', '-c', 'kill -9 `pidof clickhouse-library-bridge`'], user='root') + instance.query('DROP DICTIONARY lib_dict_c') + + +def test_server_restart_bridge_might_be_stil_alive(ch_cluster): + if instance.is_built_with_memory_sanitizer(): + pytest.skip("Memory Sanitizer cannot work with third-party shared libraries") + + create_dict_simple() + + result = instance.query('''select dictGet(lib_dict_c, 'value1', toUInt64(1));''') + assert(result.strip() == '101') + + instance.restart_clickhouse() + + result = instance.query('''select dictGet(lib_dict_c, 'value1', toUInt64(1));''') + assert(result.strip() == '101') + + instance.exec_in_container(['bash', '-c', 'kill -9 `pidof clickhouse-library-bridge`'], user='root') + instance.restart_clickhouse() + + result = instance.query('''select dictGet(lib_dict_c, 'value1', toUInt64(1));''') + assert(result.strip() == '101') + + instance.query('DROP DICTIONARY lib_dict_c') + + +def test_bridge_dies_with_parent(ch_cluster): + if instance.is_built_with_memory_sanitizer(): + pytest.skip("Memory Sanitizer cannot work with third-party shared libraries") + if instance.is_built_with_address_sanitizer(): + pytest.skip("Leak sanitizer falsely reports about a leak of 16 bytes in clickhouse-odbc-bridge") + + create_dict_simple() + result = instance.query('''select dictGet(lib_dict_c, 'value1', toUInt64(1));''') + assert(result.strip() == '101') + + clickhouse_pid = instance.get_process_pid("clickhouse server") + bridge_pid = instance.get_process_pid("library-bridge") + assert clickhouse_pid is not None + assert bridge_pid is not None + + while clickhouse_pid is not None: + try: + instance.exec_in_container(["kill", str(clickhouse_pid)], privileged=True, user='root') + except: + pass + clickhouse_pid = instance.get_process_pid("clickhouse server") + time.sleep(1) + + for i in range(30): + time.sleep(1) + bridge_pid = instance.get_process_pid("library-bridge") + if bridge_pid is None: + break + + if bridge_pid: + out = instance.exec_in_container(["gdb", "-p", str(bridge_pid), "--ex", "thread apply all bt", "--ex", "q"], + privileged=True, user='root') + logging.debug(f"Bridge is running, gdb output:\n{out}") + + assert clickhouse_pid is None + assert bridge_pid is None + instance.start_clickhouse(20) + + if __name__ == '__main__': cluster.start() input("Cluster created, press any key to destroy...") diff --git a/tests/integration/test_reloading_storage_configuration/test.py b/tests/integration/test_reloading_storage_configuration/test.py index edcba4e8a60..7fa37d0909c 100644 --- a/tests/integration/test_reloading_storage_configuration/test.py +++ b/tests/integration/test_reloading_storage_configuration/test.py @@ -7,8 +7,10 @@ import xml.etree.ElementTree as ET import helpers.client import helpers.cluster +from helpers.test_tools import TSV import pytest + cluster = helpers.cluster.ClickHouseCluster(__file__) node1 = cluster.add_instance('node1', @@ -76,6 +78,37 @@ def add_disk(node, name, path, separate_file=False): else: tree.write(os.path.join(node.config_d_dir, "storage_configuration.xml")) +def update_disk(node, name, path, keep_free_space_bytes, separate_file=False): + separate_configuration_path = os.path.join(node.config_d_dir, + "separate_configuration.xml") + + try: + if separate_file: + tree = ET.parse(separate_configuration_path) + else: + tree = ET.parse( + os.path.join(node.config_d_dir, "storage_configuration.xml")) + except: + tree = ET.ElementTree( + ET.fromstring('')) + + root = tree.getroot() + disk = root.find("storage_configuration").find("disks").find(name) + assert disk is not None + + new_path = disk.find("path") + assert new_path is not None + new_path.text = path + + new_keep_free_space_bytes = disk.find("keep_free_space_bytes") + assert new_keep_free_space_bytes is not None + new_keep_free_space_bytes.text = keep_free_space_bytes + + if separate_file: + tree.write(separate_configuration_path) + else: + tree.write(os.path.join(node.config_d_dir, "storage_configuration.xml")) + def add_policy(node, name, volumes): tree = ET.parse(os.path.join(node.config_d_dir, "storage_configuration.xml")) @@ -123,6 +156,36 @@ def test_add_disk(started_cluster): except: """""" +def test_update_disk(started_cluster): + try: + name = "test_update_disk" + engine = "MergeTree()" + + start_over() + node1.restart_clickhouse(kill=True) + time.sleep(2) + + node1.query(""" + CREATE TABLE {name} ( + d UInt64 + ) ENGINE = {engine} + ORDER BY d + SETTINGS storage_policy='jbods_with_external' + """.format(name=name, engine=engine)) + + assert node1.query("SELECT path, keep_free_space FROM system.disks where name = 'jbod2'") == TSV([ + ["/jbod2/", "10485760"]]) + + update_disk(node1, "jbod2", "/jbod2/", "20971520") + node1.query("SYSTEM RELOAD CONFIG") + + assert node1.query("SELECT path, keep_free_space FROM system.disks where name = 'jbod2'") == TSV([ + ["/jbod2/", "20971520"]]) + finally: + try: + node1.query("DROP TABLE IF EXISTS {}".format(name)) + except: + """""" def test_add_disk_to_separate_config(started_cluster): try: diff --git a/tests/integration/test_replica_is_active/__init__.py b/tests/integration/test_replica_is_active/__init__.py new file mode 100644 index 00000000000..e69de29bb2d diff --git a/tests/integration/test_replica_is_active/test.py b/tests/integration/test_replica_is_active/test.py new file mode 100644 index 00000000000..14046ea7f7d --- /dev/null +++ b/tests/integration/test_replica_is_active/test.py @@ -0,0 +1,41 @@ +import pytest +from helpers.client import QueryRuntimeException +from helpers.cluster import ClickHouseCluster + +cluster = ClickHouseCluster(__file__) +node1 = cluster.add_instance('node1', with_zookeeper=True) +node2 = cluster.add_instance('node2', with_zookeeper=True) +node3 = cluster.add_instance('node3', with_zookeeper=True) + +@pytest.fixture(scope="module") +def start_cluster(): + try: + cluster.start() + + for i, node in enumerate((node1, node2, node3)): + node_name = 'node' + str(i + 1) + node.query( + ''' + CREATE TABLE test_table(date Date, id UInt32, dummy UInt32) + ENGINE = ReplicatedMergeTree('/clickhouse/tables/test_table', '{}') + PARTITION BY date ORDER BY id + '''.format(node_name) + ) + + yield cluster + + finally: + cluster.shutdown() + + +def test_replica_is_active(start_cluster): + query_result = node1.query("select replica_is_active from system.replicas where table = 'test_table'") + assert query_result == '{\'node1\':1,\'node2\':1,\'node3\':1}\n' + + node3.stop() + query_result = node1.query("select replica_is_active from system.replicas where table = 'test_table'") + assert query_result == '{\'node1\':1,\'node2\':1,\'node3\':0}\n' + + node2.stop() + query_result = node1.query("select replica_is_active from system.replicas where table = 'test_table'") + assert query_result == '{\'node1\':1,\'node2\':0,\'node3\':0}\n' diff --git a/tests/integration/test_storage_mongodb/configs_secure/config.d/ssl_conf.xml b/tests/integration/test_storage_mongodb/configs_secure/config.d/ssl_conf.xml new file mode 100644 index 00000000000..e14aac81e17 --- /dev/null +++ b/tests/integration/test_storage_mongodb/configs_secure/config.d/ssl_conf.xml @@ -0,0 +1,8 @@ + + + + + none + + + diff --git a/tests/integration/test_storage_mongodb/test.py b/tests/integration/test_storage_mongodb/test.py index 75af909faec..415f1c1cb33 100644 --- a/tests/integration/test_storage_mongodb/test.py +++ b/tests/integration/test_storage_mongodb/test.py @@ -5,24 +5,29 @@ from helpers.client import QueryRuntimeException from helpers.cluster import ClickHouseCluster -cluster = ClickHouseCluster(__file__) -node = cluster.add_instance('node', with_mongo=True) - @pytest.fixture(scope="module") -def started_cluster(): +def started_cluster(request): try: + cluster = ClickHouseCluster(__file__) + node = cluster.add_instance('node', + main_configs=["configs_secure/config.d/ssl_conf.xml"], + with_mongo=True, + with_mongo_secure=request.param) cluster.start() yield cluster finally: cluster.shutdown() -def get_mongo_connection(started_cluster): +def get_mongo_connection(started_cluster, secure=False): connection_str = 'mongodb://root:clickhouse@localhost:{}'.format(started_cluster.mongo_port) + if secure: + connection_str += '/?tls=true&tlsAllowInvalidCertificates=true' return pymongo.MongoClient(connection_str) +@pytest.mark.parametrize('started_cluster', [False], indirect=['started_cluster']) def test_simple_select(started_cluster): mongo_connection = get_mongo_connection(started_cluster) db = mongo_connection['test'] @@ -33,6 +38,7 @@ def test_simple_select(started_cluster): data.append({'key': i, 'data': hex(i * i)}) simple_mongo_table.insert_many(data) + node = started_cluster.instances['node'] node.query( "CREATE TABLE simple_mongo_table(key UInt64, data String) ENGINE = MongoDB('mongo1:27017', 'test', 'simple_table', 'root', 'clickhouse')") @@ -42,6 +48,7 @@ def test_simple_select(started_cluster): assert node.query("SELECT data from simple_mongo_table where key = 42") == hex(42 * 42) + '\n' +@pytest.mark.parametrize('started_cluster', [False], indirect=['started_cluster']) def test_complex_data_type(started_cluster): mongo_connection = get_mongo_connection(started_cluster) db = mongo_connection['test'] @@ -52,6 +59,7 @@ def test_complex_data_type(started_cluster): data.append({'key': i, 'data': hex(i * i), 'dict': {'a': i, 'b': str(i)}}) incomplete_mongo_table.insert_many(data) + node = started_cluster.instances['node'] node.query( "CREATE TABLE incomplete_mongo_table(key UInt64, data String) ENGINE = MongoDB('mongo1:27017', 'test', 'complex_table', 'root', 'clickhouse')") @@ -61,6 +69,7 @@ def test_complex_data_type(started_cluster): assert node.query("SELECT data from incomplete_mongo_table where key = 42") == hex(42 * 42) + '\n' +@pytest.mark.parametrize('started_cluster', [False], indirect=['started_cluster']) def test_incorrect_data_type(started_cluster): mongo_connection = get_mongo_connection(started_cluster) db = mongo_connection['test'] @@ -71,6 +80,7 @@ def test_incorrect_data_type(started_cluster): data.append({'key': i, 'data': hex(i * i), 'aaaa': 'Hello'}) strange_mongo_table.insert_many(data) + node = started_cluster.instances['node'] node.query( "CREATE TABLE strange_mongo_table(key String, data String) ENGINE = MongoDB('mongo1:27017', 'test', 'strange_table', 'root', 'clickhouse')") @@ -85,3 +95,24 @@ def test_incorrect_data_type(started_cluster): with pytest.raises(QueryRuntimeException): node.query("SELECT bbbb FROM strange_mongo_table2") + + +@pytest.mark.parametrize('started_cluster', [True], indirect=['started_cluster']) +def test_secure_connection(started_cluster): + mongo_connection = get_mongo_connection(started_cluster, secure=True) + db = mongo_connection['test'] + db.add_user('root', 'clickhouse') + simple_mongo_table = db['simple_table'] + data = [] + for i in range(0, 100): + data.append({'key': i, 'data': hex(i * i)}) + simple_mongo_table.insert_many(data) + + node = started_cluster.instances['node'] + node.query( + "CREATE TABLE simple_mongo_table(key UInt64, data String) ENGINE = MongoDB('mongo1:27017', 'test', 'simple_table', 'root', 'clickhouse', 'ssl=true')") + + assert node.query("SELECT COUNT() FROM simple_mongo_table") == '100\n' + assert node.query("SELECT sum(key) FROM simple_mongo_table") == str(sum(range(0, 100))) + '\n' + + assert node.query("SELECT data from simple_mongo_table where key = 42") == hex(42 * 42) + '\n' diff --git a/tests/performance/datetime_comparison.xml b/tests/performance/datetime_comparison.xml index 2d47ded0b1a..8d7b0c8c4de 100644 --- a/tests/performance/datetime_comparison.xml +++ b/tests/performance/datetime_comparison.xml @@ -2,4 +2,5 @@ SELECT count() FROM numbers(1000000000) WHERE materialize(now()) > toString(toDateTime('2020-09-30 00:00:00')) SELECT count() FROM numbers(1000000000) WHERE materialize(now()) > toUInt32(toDateTime('2020-09-30 00:00:00')) SELECT count() FROM numbers(1000000000) WHERE materialize(now()) > toDateTime('2020-09-30 00:00:00') + SELECT count() FROM numbers(1000000000) WHERE materialize(now()) > toDate('2020-09-30 00:00:00') diff --git a/tests/performance/window_functions.xml b/tests/performance/window_functions.xml index 6be3d59e2b0..e3d30d96ec3 100644 --- a/tests/performance/window_functions.xml +++ b/tests/performance/window_functions.xml @@ -3,10 +3,6 @@ hits_100m_single - - 1 - - ... SELECT sum(n) from rich_syntax; + +-- Clear cache to avoid future errors in the logs +SYSTEM DROP DNS CACHE diff --git a/tests/queries/0_stateless/01149_zookeeper_mutation_stuck_after_replace_partition.sql b/tests/queries/0_stateless/01149_zookeeper_mutation_stuck_after_replace_partition.sql index 18508635cf9..ca8f70b3cf4 100644 --- a/tests/queries/0_stateless/01149_zookeeper_mutation_stuck_after_replace_partition.sql +++ b/tests/queries/0_stateless/01149_zookeeper_mutation_stuck_after_replace_partition.sql @@ -1,3 +1,4 @@ +set send_logs_level='error'; drop table if exists mt; drop table if exists rmt sync; diff --git a/tests/queries/0_stateless/01175_distributed_ddl_output_mode_long.sh b/tests/queries/0_stateless/01175_distributed_ddl_output_mode_long.sh index 7347fc5d626..483979d00db 100755 --- a/tests/queries/0_stateless/01175_distributed_ddl_output_mode_long.sh +++ b/tests/queries/0_stateless/01175_distributed_ddl_output_mode_long.sh @@ -14,7 +14,7 @@ function run_until_out_contains() PATTERN=$1 shift - while true + for _ in {1..20} do "$@" > "${CLICKHOUSE_TMP}/out" 2>&1 if grep -q "$PATTERN" "${CLICKHOUSE_TMP}/out" diff --git a/tests/queries/0_stateless/01244_optimize_distributed_group_by_sharding_key.sql b/tests/queries/0_stateless/01244_optimize_distributed_group_by_sharding_key.sql index 4719119165a..1dcdd795bc1 100644 --- a/tests/queries/0_stateless/01244_optimize_distributed_group_by_sharding_key.sql +++ b/tests/queries/0_stateless/01244_optimize_distributed_group_by_sharding_key.sql @@ -116,7 +116,7 @@ select distinct on (k1) k2 from remote('127.{1,2}', view(select 1 k1, 2 k2, 3 v) -- window functions select 'window functions'; -select key, sum(sum(value)) over (rows unbounded preceding) from dist_01247 group by key settings allow_experimental_window_functions=1; +select key, sum(sum(value)) over (rows unbounded preceding) from dist_01247 group by key; drop table dist_01247; drop table data_01247; diff --git a/tests/queries/0_stateless/01294_system_distributed_on_cluster.sql b/tests/queries/0_stateless/01294_system_distributed_on_cluster.sql index d56bddba3c6..525974e78ba 100644 --- a/tests/queries/0_stateless/01294_system_distributed_on_cluster.sql +++ b/tests/queries/0_stateless/01294_system_distributed_on_cluster.sql @@ -3,6 +3,7 @@ -- quirk for ON CLUSTER does not uses currentDatabase() drop database if exists db_01294; create database db_01294; +set distributed_ddl_output_mode='throw'; drop table if exists db_01294.dist_01294; create table db_01294.dist_01294 as system.one engine=Distributed(test_shard_localhost, system, one); diff --git a/tests/queries/0_stateless/01372_remote_table_function_empty_table.sql b/tests/queries/0_stateless/01372_remote_table_function_empty_table.sql index 698c323d73f..4153dc632f3 100644 --- a/tests/queries/0_stateless/01372_remote_table_function_empty_table.sql +++ b/tests/queries/0_stateless/01372_remote_table_function_empty_table.sql @@ -1 +1,4 @@ SELECT * FROM remote('127..2', 'a.'); -- { serverError 36 } + +-- Clear cache to avoid future errors in the logs +SYSTEM DROP DNS CACHE diff --git a/tests/queries/0_stateless/01415_sticking_mutations.sh b/tests/queries/0_stateless/01415_sticking_mutations.sh index 9bd0a6eeebf..2e86b6d972d 100755 --- a/tests/queries/0_stateless/01415_sticking_mutations.sh +++ b/tests/queries/0_stateless/01415_sticking_mutations.sh @@ -33,9 +33,10 @@ function check_sticky_mutations() query_result=$($CLICKHOUSE_CLIENT --query="$check_query" 2>&1) - while [ "$query_result" == "0" ] + for _ in {1..50} do query_result=$($CLICKHOUSE_CLIENT --query="$check_query" 2>&1) + if ! [ "$query_result" == "0" ]; then break; fi sleep 0.5 done ##### wait mutation to start ##### diff --git a/tests/queries/0_stateless/01551_mergetree_read_in_order_spread.reference b/tests/queries/0_stateless/01551_mergetree_read_in_order_spread.reference index 2843b305f0a..cdc595a3c57 100644 --- a/tests/queries/0_stateless/01551_mergetree_read_in_order_spread.reference +++ b/tests/queries/0_stateless/01551_mergetree_read_in_order_spread.reference @@ -9,9 +9,9 @@ ExpressionTransform (SettingQuotaAndLimits) (ReadFromMergeTree) ExpressionTransform × 4 - MergeTree 0 → 1 + MergeTreeInOrder 0 → 1 MergingSortedTransform 2 → 1 ExpressionTransform × 2 - MergeTree × 2 0 → 1 + MergeTreeInOrder × 2 0 → 1 ExpressionTransform - MergeTree 0 → 1 + MergeTreeInOrder 0 → 1 diff --git a/tests/queries/0_stateless/01568_window_functions_distributed.reference b/tests/queries/0_stateless/01568_window_functions_distributed.reference index 483e84a2bee..0b439ef759a 100644 --- a/tests/queries/0_stateless/01568_window_functions_distributed.reference +++ b/tests/queries/0_stateless/01568_window_functions_distributed.reference @@ -1,5 +1,4 @@ -- { echo } -set allow_experimental_window_functions = 1; select row_number() over (order by dummy) from (select * from remote('127.0.0.{1,2}', system, one)); 1 2 diff --git a/tests/queries/0_stateless/01568_window_functions_distributed.sql b/tests/queries/0_stateless/01568_window_functions_distributed.sql index 6f38597a7a3..5e20c57d23d 100644 --- a/tests/queries/0_stateless/01568_window_functions_distributed.sql +++ b/tests/queries/0_stateless/01568_window_functions_distributed.sql @@ -1,6 +1,4 @@ -- { echo } -set allow_experimental_window_functions = 1; - select row_number() over (order by dummy) from (select * from remote('127.0.0.{1,2}', system, one)); select row_number() over (order by dummy) from remote('127.0.0.{1,2}', system, one); diff --git a/tests/queries/0_stateless/01571_window_functions.reference b/tests/queries/0_stateless/01571_window_functions.reference index e9feb9295df..420f7575a52 100644 --- a/tests/queries/0_stateless/01571_window_functions.reference +++ b/tests/queries/0_stateless/01571_window_functions.reference @@ -1,6 +1,6 @@ -- { echo } -- Another test for window functions because the other one is too long. -set allow_experimental_window_functions = 1; + -- some craziness with a mix of materialized and unmaterialized const columns -- after merging sorted transform, that used to break the peer group detection in -- the window transform. diff --git a/tests/queries/0_stateless/01571_window_functions.sql b/tests/queries/0_stateless/01571_window_functions.sql index 56ead7a1188..4aaba19100a 100644 --- a/tests/queries/0_stateless/01571_window_functions.sql +++ b/tests/queries/0_stateless/01571_window_functions.sql @@ -1,6 +1,5 @@ -- { echo } -- Another test for window functions because the other one is too long. -set allow_experimental_window_functions = 1; -- some craziness with a mix of materialized and unmaterialized const columns -- after merging sorted transform, that used to break the peer group detection in diff --git a/tests/queries/0_stateless/01591_window_functions.reference b/tests/queries/0_stateless/01591_window_functions.reference index 20006a79e90..26e9e500c3c 100644 --- a/tests/queries/0_stateless/01591_window_functions.reference +++ b/tests/queries/0_stateless/01591_window_functions.reference @@ -1,6 +1,5 @@ -- { echo } -set allow_experimental_window_functions = 1; -- just something basic select number, count() over (partition by intDiv(number, 3) order by number rows unbounded preceding) from numbers(10); 0 1 diff --git a/tests/queries/0_stateless/01591_window_functions.sql b/tests/queries/0_stateless/01591_window_functions.sql index 59b1340bb2e..3075c1ddb46 100644 --- a/tests/queries/0_stateless/01591_window_functions.sql +++ b/tests/queries/0_stateless/01591_window_functions.sql @@ -1,7 +1,5 @@ -- { echo } -set allow_experimental_window_functions = 1; - -- just something basic select number, count() over (partition by intDiv(number, 3) order by number rows unbounded preceding) from numbers(10); diff --git a/tests/queries/0_stateless/01592_long_window_functions1.sql b/tests/queries/0_stateless/01592_long_window_functions1.sql index bb0f77ff60a..14fe3affed3 100644 --- a/tests/queries/0_stateless/01592_long_window_functions1.sql +++ b/tests/queries/0_stateless/01592_long_window_functions1.sql @@ -1,6 +1,5 @@ drop table if exists stack; -set allow_experimental_window_functions = 1; set max_insert_threads = 4; create table stack(item_id Int64, brand_id Int64, rack_id Int64, dt DateTime, expiration_dt DateTime, quantity UInt64) diff --git a/tests/queries/0_stateless/01592_window_functions.sql b/tests/queries/0_stateless/01592_window_functions.sql index 8d5033fc821..b05b04628d2 100644 --- a/tests/queries/0_stateless/01592_window_functions.sql +++ b/tests/queries/0_stateless/01592_window_functions.sql @@ -1,5 +1,3 @@ -set allow_experimental_window_functions = 1; - drop table if exists product_groups; drop table if exists products; diff --git a/tests/queries/0_stateless/01600_log_queries_with_extensive_info.reference b/tests/queries/0_stateless/01600_log_queries_with_extensive_info.reference index 453827808f4..701e72b3b8e 100644 --- a/tests/queries/0_stateless/01600_log_queries_with_extensive_info.reference +++ b/tests/queries/0_stateless/01600_log_queries_with_extensive_info.reference @@ -6,6 +6,7 @@ create database test_log_queries 18329631544365042880 Create ['test_log_queries' create table test_log_queries.logtable(i int, j int, k int) engine MergeTree order by i 14473140110122260412 Create ['test_log_queries'] ['test_log_queries.logtable'] [] insert into test_log_queries.logtable values 10533878590475998223 Insert ['test_log_queries'] ['test_log_queries.logtable'] [] select k from test_log_queries.logtable where i = 4 10551554599491277990 Select ['test_log_queries'] ['test_log_queries.logtable'] ['test_log_queries.logtable.i','test_log_queries.logtable.k'] +select k from test_log_queries.logtable where i > \'\' 10861140285495151963 Select [] [] [] select k from test_log_queries.logtable where i = 1 10551554599491277990 Select ['test_log_queries'] ['test_log_queries.logtable'] ['test_log_queries.logtable.i','test_log_queries.logtable.k'] select * from test_log_queries.logtable where i = 1 2790142879136771124 Select ['test_log_queries'] ['test_log_queries.logtable'] ['test_log_queries.logtable.i','test_log_queries.logtable.j','test_log_queries.logtable.k'] create table test_log_queries.logtable2 as test_log_queries.logtable 16326833375045356331 Create ['test_log_queries'] ['test_log_queries.logtable','test_log_queries.logtable2'] [] diff --git a/tests/queries/0_stateless/01600_log_queries_with_extensive_info.sh b/tests/queries/0_stateless/01600_log_queries_with_extensive_info.sh index a6a8f221084..6f0f1c29208 100755 --- a/tests/queries/0_stateless/01600_log_queries_with_extensive_info.sh +++ b/tests/queries/0_stateless/01600_log_queries_with_extensive_info.sh @@ -10,6 +10,10 @@ ${CLICKHOUSE_CLIENT} -q "create database test_log_queries" "--query_id=01600_log ${CLICKHOUSE_CLIENT} -q "create table test_log_queries.logtable(i int, j int, k int) engine MergeTree order by i" "--query_id=01600_log_queries_with_extensive_info_002" ${CLICKHOUSE_CLIENT} -q "insert into test_log_queries.logtable values (1,2,3), (4,5,6)" "--query_id=01600_log_queries_with_extensive_info_003" ${CLICKHOUSE_CLIENT} -q "select k from test_log_queries.logtable where i = 4" "--query_id=01600_log_queries_with_extensive_info_004" + +# exception query should also contain query_kind +${CLICKHOUSE_CLIENT} -q "select k from test_log_queries.logtable where i > ''" "--query_id=01600_log_queries_with_extensive_info_004_err" 2> /dev/null || true + ${CLICKHOUSE_CLIENT} -q "select k from test_log_queries.logtable where i = 1" "--query_id=01600_log_queries_with_extensive_info_005" ${CLICKHOUSE_CLIENT} -q "select * from test_log_queries.logtable where i = 1" "--query_id=01600_log_queries_with_extensive_info_006" ${CLICKHOUSE_CLIENT} -q "create table test_log_queries.logtable2 as test_log_queries.logtable" "--query_id=01600_log_queries_with_extensive_info_007" @@ -26,4 +30,4 @@ ${CLICKHOUSE_CLIENT} -q "drop table if exists test_log_queries.logtable3" "--que ${CLICKHOUSE_CLIENT} -q "drop database if exists test_log_queries" "--query_id=01600_log_queries_with_extensive_info_018" ${CLICKHOUSE_CLIENT} -q "system flush logs" -${CLICKHOUSE_CLIENT} -q "select columns(query, normalized_query_hash, query_kind, databases, tables, columns) apply (any) from system.query_log where current_database = currentDatabase() AND type = 'QueryFinish' and query_id like '01600_log_queries_with_extensive_info%' group by query_id order by query_id" +${CLICKHOUSE_CLIENT} -q "select columns(query, normalized_query_hash, query_kind, databases, tables, columns) apply (any) from system.query_log where current_database = currentDatabase() AND type != 'QueryStart' and query_id like '01600_log_queries_with_extensive_info%' group by query_id order by query_id" diff --git a/tests/queries/0_stateless/01656_test_query_log_factories_info.reference b/tests/queries/0_stateless/01656_test_query_log_factories_info.reference index af7feae5a38..47b3133ceca 100644 --- a/tests/queries/0_stateless/01656_test_query_log_factories_info.reference +++ b/tests/queries/0_stateless/01656_test_query_log_factories_info.reference @@ -13,6 +13,9 @@ arraySort(used_table_functions) arraySort(used_functions) ['CAST','CRC32','addDays','array','arrayFlatten','modulo','plus','pow','round','substring','tanh','toDate','toDayOfYear','toTypeName','toWeek'] +used_functions +['repeat'] + arraySort(used_data_type_families) ['Array','Int32','Nullable','String'] diff --git a/tests/queries/0_stateless/01656_test_query_log_factories_info.sql b/tests/queries/0_stateless/01656_test_query_log_factories_info.sql index 3a890ce16f9..50d50155480 100644 --- a/tests/queries/0_stateless/01656_test_query_log_factories_info.sql +++ b/tests/queries/0_stateless/01656_test_query_log_factories_info.sql @@ -1,4 +1,5 @@ SET database_atomic_wait_for_drop_and_detach_synchronously=1; +SET log_queries=1; SELECT uniqArray([1, 1, 2]), SUBSTRING('Hello, world', 7, 5), @@ -13,9 +14,16 @@ SELECT uniqArray([1, 1, 2]), countIf(toDate('2000-12-05') + number as d, toDayOfYear(d) % 2) FROM numbers(100); + +SELECT repeat('a', number) +FROM numbers(10e3) +SETTINGS max_memory_usage=4e6, max_block_size=100 +FORMAT Null; -- { serverError 241 } + SELECT ''; SYSTEM FLUSH LOGS; + SELECT arraySort(used_aggregate_functions) FROM system.query_log WHERE current_database = currentDatabase() AND type = 'QueryFinish' AND (query LIKE '%toDate(\'2000-12-05\')%') ORDER BY query_start_time DESC LIMIT 1 FORMAT TabSeparatedWithNames; @@ -36,6 +44,11 @@ FROM system.query_log WHERE current_database = currentDatabase() AND type = 'Que ORDER BY query_start_time DESC LIMIT 1 FORMAT TabSeparatedWithNames; SELECT ''; +SELECT used_functions +FROM system.query_log WHERE current_database = currentDatabase() AND type != 'QueryStart' AND (query LIKE '%repeat%') +ORDER BY query_start_time DESC LIMIT 1 FORMAT TabSeparatedWithNames; +SELECT ''; + SELECT arraySort(used_data_type_families) FROM system.query_log WHERE current_database = currentDatabase() AND type = 'QueryFinish' AND (query LIKE '%toDate(\'2000-12-05\')%') ORDER BY query_start_time DESC LIMIT 1 FORMAT TabSeparatedWithNames; diff --git a/tests/queries/0_stateless/01802_rank_corr_mann_whitney_over_window.sql b/tests/queries/0_stateless/01802_rank_corr_mann_whitney_over_window.sql index 24ee9282ac0..4b8bf0844a3 100644 --- a/tests/queries/0_stateless/01802_rank_corr_mann_whitney_over_window.sql +++ b/tests/queries/0_stateless/01802_rank_corr_mann_whitney_over_window.sql @@ -1,7 +1,5 @@ DROP TABLE IF EXISTS 01802_empsalary; -SET allow_experimental_window_functions=1; - CREATE TABLE 01802_empsalary ( `depname` LowCardinality(String), diff --git a/tests/queries/0_stateless/01861_explain_pipeline.reference b/tests/queries/0_stateless/01861_explain_pipeline.reference index 9d62fb9f6b8..63ba55f5a04 100644 --- a/tests/queries/0_stateless/01861_explain_pipeline.reference +++ b/tests/queries/0_stateless/01861_explain_pipeline.reference @@ -5,7 +5,7 @@ ExpressionTransform ExpressionTransform ReplacingSorted 2 → 1 ExpressionTransform × 2 - MergeTree × 2 0 → 1 + MergeTreeInOrder × 2 0 → 1 0 0 1 1 2 2 @@ -22,4 +22,4 @@ ExpressionTransform × 2 Copy × 2 1 → 2 AddingSelector × 2 ExpressionTransform × 2 - MergeTree × 2 0 → 1 + MergeTreeInOrder × 2 0 → 1 diff --git a/tests/queries/0_stateless/01926_order_by_desc_limit.reference b/tests/queries/0_stateless/01926_order_by_desc_limit.reference new file mode 100644 index 00000000000..6ed281c757a --- /dev/null +++ b/tests/queries/0_stateless/01926_order_by_desc_limit.reference @@ -0,0 +1,2 @@ +1 +1 diff --git a/tests/queries/0_stateless/01926_order_by_desc_limit.sql b/tests/queries/0_stateless/01926_order_by_desc_limit.sql new file mode 100644 index 00000000000..7ea102e11e9 --- /dev/null +++ b/tests/queries/0_stateless/01926_order_by_desc_limit.sql @@ -0,0 +1,21 @@ +DROP TABLE IF EXISTS order_by_desc; + +CREATE TABLE order_by_desc (u UInt32, s String) +ENGINE MergeTree ORDER BY u PARTITION BY u % 100 +SETTINGS index_granularity = 1024; + +INSERT INTO order_by_desc SELECT number, repeat('a', 1024) FROM numbers(1024 * 300); +OPTIMIZE TABLE order_by_desc FINAL; + +SELECT s FROM order_by_desc ORDER BY u DESC LIMIT 10 FORMAT Null +SETTINGS max_memory_usage = '400M'; + +SELECT s FROM order_by_desc ORDER BY u LIMIT 10 FORMAT Null +SETTINGS max_memory_usage = '400M'; + +SYSTEM FLUSH LOGS; + +SELECT read_rows < 110000 FROM system.query_log +WHERE type = 'QueryFinish' AND current_database = currentDatabase() +AND event_time > now() - INTERVAL 10 SECOND +AND lower(query) LIKE lower('SELECT s FROM order_by_desc ORDER BY u%'); diff --git a/tests/queries/0_stateless/01950_kill_large_group_by_query.reference b/tests/queries/0_stateless/01950_kill_large_group_by_query.reference new file mode 100644 index 00000000000..1602d6587ad --- /dev/null +++ b/tests/queries/0_stateless/01950_kill_large_group_by_query.reference @@ -0,0 +1,2 @@ +finished test_01948_tcp_default default SELECT * FROM\n (\n SELECT a.name as n\n FROM\n (\n SELECT \'Name\' as name, number FROM system.numbers LIMIT 2000000\n ) AS a,\n (\n SELECT \'Name\' as name, number FROM system.numbers LIMIT 2000000\n ) as b\n GROUP BY n\n )\n LIMIT 20\n FORMAT Null +finished test_01948_http_default default SELECT * FROM\n (\n SELECT a.name as n\n FROM\n (\n SELECT \'Name\' as name, number FROM system.numbers LIMIT 2000000\n ) AS a,\n (\n SELECT \'Name\' as name, number FROM system.numbers LIMIT 2000000\n ) as b\n GROUP BY n\n )\n LIMIT 20\n FORMAT Null diff --git a/tests/queries/0_stateless/01950_kill_large_group_by_query.sh b/tests/queries/0_stateless/01950_kill_large_group_by_query.sh new file mode 100755 index 00000000000..465b923187e --- /dev/null +++ b/tests/queries/0_stateless/01950_kill_large_group_by_query.sh @@ -0,0 +1,54 @@ +#!/usr/bin/env bash + +CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +# shellcheck source=../shell_config.sh +. "$CURDIR"/../shell_config.sh + +set -e -o pipefail + +function wait_for_query_to_start() +{ + while [[ $($CLICKHOUSE_CLIENT -q "SELECT count() FROM system.processes WHERE query_id = '$1'") == 0 ]]; do sleep 0.1; done +} + + +# TCP CLIENT + +$CLICKHOUSE_CLIENT --max_execution_time 10 --query_id "test_01948_tcp_$CLICKHOUSE_DATABASE" -q \ + "SELECT * FROM + ( + SELECT a.name as n + FROM + ( + SELECT 'Name' as name, number FROM system.numbers LIMIT 2000000 + ) AS a, + ( + SELECT 'Name' as name, number FROM system.numbers LIMIT 2000000 + ) as b + GROUP BY n + ) + LIMIT 20 + FORMAT Null" > /dev/null 2>&1 & +wait_for_query_to_start "test_01948_tcp_$CLICKHOUSE_DATABASE" +$CLICKHOUSE_CLIENT --max_execution_time 10 -q "KILL QUERY WHERE query_id = 'test_01948_tcp_$CLICKHOUSE_DATABASE' SYNC" + + +# HTTP CLIENT + +${CLICKHOUSE_CURL_COMMAND} -q --max-time 10 -sS "$CLICKHOUSE_URL&query_id=test_01948_http_$CLICKHOUSE_DATABASE" -d \ + "SELECT * FROM + ( + SELECT a.name as n + FROM + ( + SELECT 'Name' as name, number FROM system.numbers LIMIT 2000000 + ) AS a, + ( + SELECT 'Name' as name, number FROM system.numbers LIMIT 2000000 + ) as b + GROUP BY n + ) + LIMIT 20 + FORMAT Null" > /dev/null 2>&1 & +wait_for_query_to_start "test_01948_http_$CLICKHOUSE_DATABASE" +$CLICKHOUSE_CURL --max-time 10 -sS "$CLICKHOUSE_URL" -d "KILL QUERY WHERE query_id = 'test_01948_http_$CLICKHOUSE_DATABASE' SYNC" diff --git a/tests/queries/0_stateless/02001_add_default_database_to_system_users.reference b/tests/queries/0_stateless/02001_add_default_database_to_system_users.reference new file mode 100644 index 00000000000..bec3a35ee8b --- /dev/null +++ b/tests/queries/0_stateless/02001_add_default_database_to_system_users.reference @@ -0,0 +1 @@ +system diff --git a/tests/queries/0_stateless/02001_add_default_database_to_system_users.sql b/tests/queries/0_stateless/02001_add_default_database_to_system_users.sql new file mode 100644 index 00000000000..b006f9acb22 --- /dev/null +++ b/tests/queries/0_stateless/02001_add_default_database_to_system_users.sql @@ -0,0 +1,3 @@ +create user if not exists u_02001 default database system; +select default_database from system.users where name = 'u_02001'; +drop user if exists u_02001; diff --git a/tests/queries/0_stateless/02001_compress_output_file.reference b/tests/queries/0_stateless/02001_compress_output_file.reference new file mode 100644 index 00000000000..6f51dfc24e1 --- /dev/null +++ b/tests/queries/0_stateless/02001_compress_output_file.reference @@ -0,0 +1,2 @@ +Hello, World! From client. +Hello, World! From local. diff --git a/tests/queries/0_stateless/02001_compress_output_file.sh b/tests/queries/0_stateless/02001_compress_output_file.sh new file mode 100755 index 00000000000..11df227cc14 --- /dev/null +++ b/tests/queries/0_stateless/02001_compress_output_file.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash + +CURDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +# shellcheck source=../shell_config.sh +. "$CURDIR"/../shell_config.sh + +set -e + +[ -e "${CLICKHOUSE_TMP}"/test_compression_of_output_file_from_client.gz ] && rm "${CLICKHOUSE_TMP}"/test_compression_of_output_file_from_client.gz + +${CLICKHOUSE_CLIENT} --query "SELECT * FROM (SELECT 'Hello, World! From client.') INTO OUTFILE '${CLICKHOUSE_TMP}/test_compression_of_output_file_from_client.gz'" +gunzip ${CLICKHOUSE_TMP}/test_compression_of_output_file_from_client.gz +cat ${CLICKHOUSE_TMP}/test_compression_of_output_file_from_client + +rm -f "${CLICKHOUSE_TMP}/test_compression_of_output_file_from_client" + +[ -e "${CLICKHOUSE_TMP}"/test_compression_of_output_file_from_local.gz ] && rm "${CLICKHOUSE_TMP}"/test_compression_of_output_file_from_local.gz + +${CLICKHOUSE_LOCAL} --query "SELECT * FROM (SELECT 'Hello, World! From local.') INTO OUTFILE '${CLICKHOUSE_TMP}/test_compression_of_output_file_from_local.gz'" +gunzip ${CLICKHOUSE_TMP}/test_compression_of_output_file_from_local.gz +cat ${CLICKHOUSE_TMP}/test_compression_of_output_file_from_local + +rm -f "${CLICKHOUSE_TMP}/test_compression_of_output_file_from_local" diff --git a/tests/queries/0_stateless/02001_hostname_test.reference b/tests/queries/0_stateless/02001_hostname_test.reference new file mode 100644 index 00000000000..da8a2d07eab --- /dev/null +++ b/tests/queries/0_stateless/02001_hostname_test.reference @@ -0,0 +1,2 @@ +localhost +localhost 2 diff --git a/tests/queries/0_stateless/02001_hostname_test.sql b/tests/queries/0_stateless/02001_hostname_test.sql new file mode 100644 index 00000000000..a8c7a8dab0c --- /dev/null +++ b/tests/queries/0_stateless/02001_hostname_test.sql @@ -0,0 +1,2 @@ +select hostname(); +select hostName() h, count() from cluster(test_cluster_two_shards, system.one) group by h; diff --git a/tests/queries/0_stateless/02002_global_subqueries_subquery_or_table_name.reference b/tests/queries/0_stateless/02002_global_subqueries_subquery_or_table_name.reference new file mode 100644 index 00000000000..e69de29bb2d diff --git a/tests/queries/0_stateless/02002_global_subqueries_subquery_or_table_name.sql b/tests/queries/0_stateless/02002_global_subqueries_subquery_or_table_name.sql new file mode 100644 index 00000000000..1b24617569c --- /dev/null +++ b/tests/queries/0_stateless/02002_global_subqueries_subquery_or_table_name.sql @@ -0,0 +1,5 @@ +SELECT + cityHash64(number GLOBAL IN (NULL, -2147483648, -9223372036854775808), nan, 1024, NULL, NULL, 1.000100016593933, NULL), + (NULL, cityHash64(inf, -2147483648, NULL, NULL, 10.000100135803223), cityHash64(1.1754943508222875e-38, NULL, NULL, NULL), 2147483647) +FROM cluster(test_cluster_two_shards_localhost, numbers((NULL, cityHash64(0., 65536, NULL, NULL, 10000000000., NULL), 0) GLOBAL IN (some_identifier), 65536)) +WHERE number GLOBAL IN [1025] --{serverError 284} diff --git a/tests/queries/skip_list.json b/tests/queries/skip_list.json index 3eede6783f8..d40d9a940d0 100644 --- a/tests/queries/skip_list.json +++ b/tests/queries/skip_list.json @@ -119,16 +119,14 @@ "memory_tracking", "memory_usage", "live_view", - "01181_db_atomic_drop_on_cluster", - "01175_distributed_ddl_output_mode", - "01415_sticking_mutations", - "00980_zookeeper_merge_tree_alter_settings", "01148_zookeeper_path_macros_unfolding", - "01294_system_distributed_on_cluster", "01269_create_with_null", "01451_replicated_detach_drop_and_quorum", "01188_attach_table_from_path", - "01149_zookeeper_mutation_stuck_after_replace_partition", + /// ON CLUSTER is not allowed + "01181_db_atomic_drop_on_cluster", + "01175_distributed_ddl_output_mode", + "01415_sticking_mutations", /// user_files "01721_engine_file_truncate_on_insert", /// Fails due to additional replicas or shards @@ -158,6 +156,8 @@ "00152_insert_different_granularity", "00054_merge_tree_partitions", "01781_merge_tree_deduplication", + "00980_zookeeper_merge_tree_alter_settings", + "00980_merge_alter_settings", /// Old syntax is not allowed "01062_alter_on_mutataion_zookeeper", "00925_zookeeper_empty_replicated_merge_tree_optimize_final", @@ -175,10 +175,11 @@ /// Does not support renaming of multiple tables in single query "00634_rename_view", "00140_rename", + /// Different query_id + "01943_query_id_check", /// Requires investigation "00953_zookeeper_suetin_deduplication_bug", "01783_http_chunk_size", - "01943_query_id_check", "00166_explain_estimate" ], "polymorphic-parts": [ @@ -525,6 +526,7 @@ "01902_table_function_merge_db_repr", "01946_test_zstd_decompression_with_escape_sequence_at_the_end_of_buffer", "01946_test_wrong_host_name_access", - "01213_alter_rename_with_default_zookeeper" /// Warning: Removing leftovers from table. + "01213_alter_rename_with_default_zookeeper", /// Warning: Removing leftovers from table. + "02001_add_default_database_to_system_users" ///create user ] } diff --git a/tests/testflows/window_functions/tests/common.py b/tests/testflows/window_functions/tests/common.py index 3a6ac95bd9b..3ed4f794ada 100644 --- a/tests/testflows/window_functions/tests/common.py +++ b/tests/testflows/window_functions/tests/common.py @@ -386,23 +386,3 @@ def create_table(self, name, statement, on_cluster=False): node.query(f"DROP TABLE IF EXISTS {name} ON CLUSTER {on_cluster}") else: node.query(f"DROP TABLE IF EXISTS {name}") - -@TestStep(Given) -def allow_experimental_window_functions(self): - """Set allow_experimental_window_functions = 1 - """ - setting = ("allow_experimental_window_functions", 1) - default_query_settings = None - - try: - with By("adding allow_experimental_window_functions to the default query settings"): - default_query_settings = getsattr(current().context, "default_query_settings", []) - default_query_settings.append(setting) - yield - finally: - with Finally("I remove allow_experimental_window_functions from the default query settings"): - if default_query_settings: - try: - default_query_settings.pop(default_query_settings.index(setting)) - except ValueError: - pass diff --git a/tests/testflows/window_functions/tests/feature.py b/tests/testflows/window_functions/tests/feature.py index 124660e8802..e9f137360b8 100755 --- a/tests/testflows/window_functions/tests/feature.py +++ b/tests/testflows/window_functions/tests/feature.py @@ -17,9 +17,6 @@ def feature(self, distributed, node="clickhouse1"): self.context.distributed = distributed self.context.node = self.context.cluster.node(node) - with Given("I allow experimental window functions"): - allow_experimental_window_functions() - with And("employee salary table"): empsalary_table(distributed=distributed) diff --git a/utils/changelog/README.md b/utils/changelog/README.md index ff3ac39f632..69a190fdedc 100644 --- a/utils/changelog/README.md +++ b/utils/changelog/README.md @@ -1,5 +1,14 @@ ## Generate changelog +Generate github token: +* https://github.com/settings/tokens - keep all checkboxes unchecked, no scopes need to be enabled. + +Dependencies: +``` + apt-get install git curl jq python3 python3-fuzzywuzzy +``` + + Usage example: ```