StorageKafka (topic-name): [rdk:CONFWARN] [thrd:app]: Configuration property group.id is a consumer property and will be ignored by this producer instance ```. [#37228](https://github.com/ClickHouse/ClickHouse/pull/37228) ([Mark Andreev](https://github.com/mrk-andreev)).
+* fix MySQL database engine to compatible with binary(0) dataType. [#37232](https://github.com/ClickHouse/ClickHouse/pull/37232) ([zzsmdfj](https://github.com/zzsmdfj)).
+* Fix execution of mutations in tables, in which there exist columns of type `Object`. Using subcolumns of type `Object` in `WHERE` expression of `UPDATE` or `DELETE` queries is now allowed yet, as well as manipulating (`DROP`, `MODIFY`) of separate subcolumns. Fixes [#37205](https://github.com/ClickHouse/ClickHouse/issues/37205). [#37266](https://github.com/ClickHouse/ClickHouse/pull/37266) ([Anton Popov](https://github.com/CurtizJ)).
+* Fix Nullable(String) to Nullable(Bool/IPv4/IPv6) conversion Closes [#37221](https://github.com/ClickHouse/ClickHouse/issues/37221). [#37270](https://github.com/ClickHouse/ClickHouse/pull/37270) ([Kruglov Pavel](https://github.com/Avogar)).
+* Fix system.opentelemetry_span_log attribute.values alias to values instead of keys. [#37275](https://github.com/ClickHouse/ClickHouse/pull/37275) ([Aleksandr Razumov](https://github.com/ernado)).
+* Fix possible deadlock in OvercommitTracker during logging. cc @alesapin @tavplubix Fixes [#37272](https://github.com/ClickHouse/ClickHouse/issues/37272). [#37299](https://github.com/ClickHouse/ClickHouse/pull/37299) ([Dmitry Novik](https://github.com/novikd)).
+
+#### Bug Fix (user-visible misbehaviour in official stable or prestable release)
+
+* - fix substring function range error length when `offset` and `length` is negative constant and `s` is not constant. [#33861](https://github.com/ClickHouse/ClickHouse/pull/33861) ([RogerYK](https://github.com/RogerYK)).
+* Accidentally ZSTD support for Arrow was not being built. This fixes [#35283](https://github.com/ClickHouse/ClickHouse/issues/35283). [#35486](https://github.com/ClickHouse/ClickHouse/pull/35486) ([Sean Lafferty](https://github.com/seanlaff)).
+* Fix ALTER DROP COLUMN of nested column with compact parts (i.e. `ALTER TABLE x DROP COLUMN n`, when there is column `n.d`). [#35797](https://github.com/ClickHouse/ClickHouse/pull/35797) ([Azat Khuzhin](https://github.com/azat)).
+* Fix insertion of complex JSONs with nested arrays to columns of type `Object`. [#36077](https://github.com/ClickHouse/ClickHouse/pull/36077) ([Anton Popov](https://github.com/CurtizJ)).
+* Queries with aliases inside special operators returned parsing error (was broken in 22.1). Example: `SELECT substring('test' AS t, 1, 1)`. [#36167](https://github.com/ClickHouse/ClickHouse/pull/36167) ([Maksim Kita](https://github.com/kitaisreal)).
+* - Fix assertion in JOIN, close [#36199](https://github.com/ClickHouse/ClickHouse/issues/36199). [#36201](https://github.com/ClickHouse/ClickHouse/pull/36201) ([Vladimir C](https://github.com/vdimir)).
+* Fix dictionary reload for `ClickHouseDictionarySource` if it contains scalar subqueries. [#36390](https://github.com/ClickHouse/ClickHouse/pull/36390) ([lthaooo](https://github.com/lthaooo)).
+* Fix nullptr dereference in JOIN and COLUMNS matcher. This fixes [#36416](https://github.com/ClickHouse/ClickHouse/issues/36416) . This is for https://github.com/ClickHouse/ClickHouse/pull/36417. [#36430](https://github.com/ClickHouse/ClickHouse/pull/36430) ([Amos Bird](https://github.com/amosbird)).
+* Fix bug in s3Cluster schema inference that let to the fact that not all data was read in the select from s3Cluster. The bug appeared in https://github.com/ClickHouse/ClickHouse/pull/35544. [#36434](https://github.com/ClickHouse/ClickHouse/pull/36434) ([Kruglov Pavel](https://github.com/Avogar)).
+* Server might fail to start if it cannot resolve hostname of external ClickHouse dictionary. It's fixed. Fixes [#36451](https://github.com/ClickHouse/ClickHouse/issues/36451). [#36463](https://github.com/ClickHouse/ClickHouse/pull/36463) ([Alexander Tokmakov](https://github.com/tavplubix)).
+* This code segment can prove bug. ``` int main() { RangeGenerator g{1230, 100}; std::cout << g.totalRanges() << std::endl; int count = 0; while(g.nextRange()) ++count; std::cout << "count:" << count << std::endl; return 0; }. [#36469](https://github.com/ClickHouse/ClickHouse/pull/36469) ([李扬](https://github.com/taiyang-li)).
+* Fix clickhouse-benchmark json report results. [#36473](https://github.com/ClickHouse/ClickHouse/pull/36473) ([Tian Xinhui](https://github.com/xinhuitian)).
+* Add missing enum values in system.session_log table. Closes [#36474](https://github.com/ClickHouse/ClickHouse/issues/36474). [#36480](https://github.com/ClickHouse/ClickHouse/pull/36480) ([Memo](https://github.com/Joeywzr)).
+* Fix possible exception with unknown packet from server in client. [#36481](https://github.com/ClickHouse/ClickHouse/pull/36481) ([Kseniia Sumarokova](https://github.com/kssenii)).
+* Fix usage of executable user defined functions in GROUP BY. Before executable user defined functions cannot be used as expressions in GROUP BY. Closes [#36448](https://github.com/ClickHouse/ClickHouse/issues/36448). [#36486](https://github.com/ClickHouse/ClickHouse/pull/36486) ([Maksim Kita](https://github.com/kitaisreal)).
+* close [#33906](https://github.com/ClickHouse/ClickHouse/issues/33906). [#36489](https://github.com/ClickHouse/ClickHouse/pull/36489) ([awakeljw](https://github.com/awakeljw)).
+* Fix hostname sanity checks for Keeper cluster configuration. Add `keeper_server.host_checks_enabled` config to enable/disable those checks. [#36492](https://github.com/ClickHouse/ClickHouse/pull/36492) ([Antonio Andelic](https://github.com/antonio2368)).
+* Fix offset update ReadBufferFromEncryptedFile, which could cause undefined behaviour. [#36493](https://github.com/ClickHouse/ClickHouse/pull/36493) ([Kseniia Sumarokova](https://github.com/kssenii)).
+* - Fix potential error with literals in `WHERE` for join queries. Close [#36279](https://github.com/ClickHouse/ClickHouse/issues/36279). [#36542](https://github.com/ClickHouse/ClickHouse/pull/36542) ([Vladimir C](https://github.com/vdimir)).
+* Fix `Missing column` exception which could happen while using `INTERPOLATE` with `ENGINE = MergeTree` table. [#36549](https://github.com/ClickHouse/ClickHouse/pull/36549) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)).
+* Fix format crash when default expression follow EPHEMERAL not literal. Closes [#36618](https://github.com/ClickHouse/ClickHouse/issues/36618). [#36633](https://github.com/ClickHouse/ClickHouse/pull/36633) ([flynn](https://github.com/ucasfl)).
+* Fix merges of wide parts with type `Object`. [#36637](https://github.com/ClickHouse/ClickHouse/pull/36637) ([Anton Popov](https://github.com/CurtizJ)).
+* Fixed parsing of query settings in `CREATE` query when engine is not specified. Fixes https://github.com/ClickHouse/ClickHouse/pull/34187#issuecomment-1103812419. [#36642](https://github.com/ClickHouse/ClickHouse/pull/36642) ([Alexander Tokmakov](https://github.com/tavplubix)).
+* Fix possible heap-use-after-free in schema inference. Closes [#36661](https://github.com/ClickHouse/ClickHouse/issues/36661). [#36679](https://github.com/ClickHouse/ClickHouse/pull/36679) ([Kruglov Pavel](https://github.com/Avogar)).
+* Fix server restart if cache configuration changed. [#36685](https://github.com/ClickHouse/ClickHouse/pull/36685) ([Kseniia Sumarokova](https://github.com/kssenii)).
+* In the previous [PR](https://github.com/ClickHouse/ClickHouse/pull/36376), I found that testing **(stateless tests, flaky check (address, actions))** is timeout. Moreover, testing locally can also trigger unstable system deadlocks. This problem still exists when using the latest source code of master. [#36697](https://github.com/ClickHouse/ClickHouse/pull/36697) ([Han Shukai](https://github.com/KinderRiven)).
+* Fix server reload on port change (do not wait for current connections from query context). [#36700](https://github.com/ClickHouse/ClickHouse/pull/36700) ([Azat Khuzhin](https://github.com/azat)).
+* Fix vertical merges in wide parts. Previously an exception `There is no column` can be thrown during merge. [#36707](https://github.com/ClickHouse/ClickHouse/pull/36707) ([Anton Popov](https://github.com/CurtizJ)).
+* During the [test](https://s3.amazonaws.com/clickhouse-test-reports/36376/1cb1c7275cb53769ab826772db9b71361bb3e413/stress_test__thread__actions_/clickhouse-server.clean.log) in [PR](https://github.com/ClickHouse/ClickHouse/pull/36376), I found that the one cache class was initialized twice, it throws a exception. Although the cause of this problem is not clear, there should be code logic of repeatedly loading disk in ClickHouse, so we need to make special judgment for this situation. [#36737](https://github.com/ClickHouse/ClickHouse/pull/36737) ([Han Shukai](https://github.com/KinderRiven)).
+* Fix a bug of `groupBitmapAndState`/`groupBitmapOrState`/`groupBitmapXorState` on distributed table. [#36739](https://github.com/ClickHouse/ClickHouse/pull/36739) ([Zhang Yifan](https://github.com/zhangyifan27)).
+* Fix timeouts in Hedged requests. Connection hang right after sending remote query could lead to eternal waiting. [#36749](https://github.com/ClickHouse/ClickHouse/pull/36749) ([Kruglov Pavel](https://github.com/Avogar)).
+* Fix insertion to columns of type `Object` from multiple files, e.g. via table function `file` with globs. [#36762](https://github.com/ClickHouse/ClickHouse/pull/36762) ([Anton Popov](https://github.com/CurtizJ)).
+* Fix some issues with async reads from remote filesystem which happened when reading low cardinality. [#36763](https://github.com/ClickHouse/ClickHouse/pull/36763) ([Kseniia Sumarokova](https://github.com/kssenii)).
+* Fix creation of tables with `flatten_nested = 0`. Previously unflattened `Nested` columns could be flattened after server restart. [#36803](https://github.com/ClickHouse/ClickHouse/pull/36803) ([Anton Popov](https://github.com/CurtizJ)).
+* Fix incorrect cast in cached buffer from remote fs. [#36809](https://github.com/ClickHouse/ClickHouse/pull/36809) ([Kseniia Sumarokova](https://github.com/kssenii)).
+* Remove function `groupArraySorted` which has a bug. [#36822](https://github.com/ClickHouse/ClickHouse/pull/36822) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
+* Fix fire in window view with hop window [#34044](https://github.com/ClickHouse/ClickHouse/issues/34044). [#36861](https://github.com/ClickHouse/ClickHouse/pull/36861) ([vxider](https://github.com/Vxider)).
+* Fix `current_size` count in cache. [#36887](https://github.com/ClickHouse/ClickHouse/pull/36887) ([Kseniia Sumarokova](https://github.com/kssenii)).
+* Fix incorrect query result when doing constant aggregation. This fixes [#36728](https://github.com/ClickHouse/ClickHouse/issues/36728) . [#36888](https://github.com/ClickHouse/ClickHouse/pull/36888) ([Amos Bird](https://github.com/amosbird)).
+* Fix bug in clickhouse-keeper which can lead to corrupted compressed log files in case of small load and restarts. [#36910](https://github.com/ClickHouse/ClickHouse/pull/36910) ([alesapin](https://github.com/alesapin)).
+* Fix bugs when using multiple columns in WindowView by adding converting actions to make it possible to call`writeIntoWindowView` with a slightly different schema. [#36928](https://github.com/ClickHouse/ClickHouse/pull/36928) ([vxider](https://github.com/Vxider)).
+* Fix issue: [#36671](https://github.com/ClickHouse/ClickHouse/issues/36671). [#36929](https://github.com/ClickHouse/ClickHouse/pull/36929) ([李扬](https://github.com/taiyang-li)).
+* Fix stuck when dropping source table in WindowView. Closes [#35678](https://github.com/ClickHouse/ClickHouse/issues/35678). [#36967](https://github.com/ClickHouse/ClickHouse/pull/36967) ([vxider](https://github.com/Vxider)).
+* Fixed logical error on `TRUNCATE` query in `Replicated` database. Fixes [#33747](https://github.com/ClickHouse/ClickHouse/issues/33747). [#36976](https://github.com/ClickHouse/ClickHouse/pull/36976) ([Alexander Tokmakov](https://github.com/tavplubix)).
+* Fix sending external tables data in HedgedConnections with max_parallel_replicas != 1. [#36981](https://github.com/ClickHouse/ClickHouse/pull/36981) ([Kruglov Pavel](https://github.com/Avogar)).
+* Fixed problem with infs in `quantileTDigest`. Fixes [#32107](https://github.com/ClickHouse/ClickHouse/issues/32107). [#37021](https://github.com/ClickHouse/ClickHouse/pull/37021) ([Vladimir Chebotarev](https://github.com/excitoon)).
+* Fix LowCardinality->ArrowDictionary invalid output when type of indexes is not UInt8. Closes [#36832](https://github.com/ClickHouse/ClickHouse/issues/36832). [#37043](https://github.com/ClickHouse/ClickHouse/pull/37043) ([Kruglov Pavel](https://github.com/Avogar)).
+* Fix in-order `GROUP BY` (`optimize_aggregation_in_order=1`) with `*Array` (`groupArrayArray`/...) aggregate functions. [#37046](https://github.com/ClickHouse/ClickHouse/pull/37046) ([Azat Khuzhin](https://github.com/azat)).
+* Fixed performance degradation of some INSERT SELECT queries with implicit aggregation. Fixes [#36792](https://github.com/ClickHouse/ClickHouse/issues/36792). [#37047](https://github.com/ClickHouse/ClickHouse/pull/37047) ([Alexander Tokmakov](https://github.com/tavplubix)).
+* Fix optimize_aggregation_in_order with prefix GROUP BY and *Array aggregate functions. [#37050](https://github.com/ClickHouse/ClickHouse/pull/37050) ([Azat Khuzhin](https://github.com/azat)).
+
+#### NO CL ENTRY
+
+* NO CL ENTRY: 'Revert "Minor refactor to prefer C++ Standard Algorithms"'. [#36511](https://github.com/ClickHouse/ClickHouse/pull/36511) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
+* NO CL ENTRY: 'Revert "Strict taskstats parser"'. [#36591](https://github.com/ClickHouse/ClickHouse/pull/36591) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
+* NO CL ENTRY: 'Revert "Translate docs/zh/sql-reference/data-types/map.md"'. [#36594](https://github.com/ClickHouse/ClickHouse/pull/36594) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
+* NO CL ENTRY: 'Revert "Update setting.md"'. [#36595](https://github.com/ClickHouse/ClickHouse/pull/36595) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
+* NO CL ENTRY: 'Documentation: Add a missing **ESTIMATE** in explain syntax'. [#36717](https://github.com/ClickHouse/ClickHouse/pull/36717) ([小蝌蚪](https://github.com/kayhaw)).
+* NO CL ENTRY: '[Snyk] Security upgrade numpy from 1.16.6 to 1.22.2'. [#36729](https://github.com/ClickHouse/ClickHouse/pull/36729) ([Mikhail f. Shiryaev](https://github.com/Felixoid)).
+* NO CL ENTRY: 'Translate playground.md to Chinese'. [#36821](https://github.com/ClickHouse/ClickHouse/pull/36821) ([小蝌蚪](https://github.com/kayhaw)).
+* NO CL ENTRY: 'Revert "Memory overcommit: continue query execution if memory is available"'. [#36858](https://github.com/ClickHouse/ClickHouse/pull/36858) ([alesapin](https://github.com/alesapin)).
+* NO CL ENTRY: 'Revert "Revert "Memory overcommit: continue query execution if memory is available""'. [#36859](https://github.com/ClickHouse/ClickHouse/pull/36859) ([Dmitry Novik](https://github.com/novikd)).
+* NO CL ENTRY: 'Revert "BLAKE3 hash function documentation"'. [#37092](https://github.com/ClickHouse/ClickHouse/pull/37092) ([Rich Raposa](https://github.com/rfraposa)).
+* NO CL ENTRY: 'Revert "Remove height restrictions from the query div in play web tool."'. [#37261](https://github.com/ClickHouse/ClickHouse/pull/37261) ([Alexey Milovidov](https://github.com/alexey-milovidov)).
+
diff --git a/docs/en/development/cmake-in-clickhouse.md b/docs/en/development/cmake-in-clickhouse.md
index 14b98b136b3..65d280df902 100644
--- a/docs/en/development/cmake-in-clickhouse.md
+++ b/docs/en/development/cmake-in-clickhouse.md
@@ -420,12 +420,6 @@ Note that ClickHouse uses forks of these libraries, see https://github.com/Click
| Using system libs can cause a lot of warnings in includes (on macro expansion). |
-WEVERYTHING |
-ON |
-Enable -Weverything option with some exceptions. |
-Add some warnings that are not available even with -Wall -Wextra -Wpedantic. Intended for exploration of new compiler warnings that may be found useful. Applies to clang only |
-
-
WITH_COVERAGE |
OFF |
Profile the resulting binary/binaries |
diff --git a/docs/en/engines/table-engines/mergetree-family/mergetree.md b/docs/en/engines/table-engines/mergetree-family/mergetree.md
index 1029cceb28a..d59b07b5dd6 100644
--- a/docs/en/engines/table-engines/mergetree-family/mergetree.md
+++ b/docs/en/engines/table-engines/mergetree-family/mergetree.md
@@ -669,6 +669,7 @@ Storage policies configuration markup:
disk_name_from_disks_configuration
1073741824
+ round_robin
@@ -695,6 +696,8 @@ Tags:
- `max_data_part_size_bytes` — the maximum size of a part that can be stored on any of the volume’s disks. If the a size of a merged part estimated to be bigger than `max_data_part_size_bytes` then this part will be written to a next volume. Basically this feature allows to keep new/small parts on a hot (SSD) volume and move them to a cold (HDD) volume when they reach large size. Do not use this setting if your policy has only one volume.
- `move_factor` — when the amount of available space gets lower than this factor, data automatically starts to move on the next volume if any (by default, 0.1). ClickHouse sorts existing parts by size from largest to smallest (in descending order) and selects parts with the total size that is sufficient to meet the `move_factor` condition. If the total size of all parts is insufficient, all parts will be moved.
- `prefer_not_to_merge` — Disables merging of data parts on this volume. When this setting is enabled, merging data on this volume is not allowed. This allows controlling how ClickHouse works with slow disks.
+- `perform_ttl_move_on_insert` — Disables TTL move on data part INSERT. By default if we insert a data part that already expired by the TTL move rule it immediately goes to a volume/disk declared in move rule. This can significantly slowdown insert in case if destination volume/disk is slow (e.g. S3).
+- `load_balancing` - Policy for disk balancing, `round_robin` or `least_used`.
Cofiguration examples:
@@ -724,7 +727,7 @@ Cofiguration examples:
0.2
-
+
jbod1
diff --git a/docs/en/getting-started/install.md b/docs/en/getting-started/install.md
index f31a78bc1c4..12775749a25 100644
--- a/docs/en/getting-started/install.md
+++ b/docs/en/getting-started/install.md
@@ -238,7 +238,7 @@ To start the server as a daemon, run:
$ sudo clickhouse start
```
-There are also another ways to run ClickHouse:
+There are also other ways to run ClickHouse:
``` bash
$ sudo service clickhouse-server start
diff --git a/docs/en/interfaces/formats.md b/docs/en/interfaces/formats.md
index e382bbcddd8..31f948cbb00 100644
--- a/docs/en/interfaces/formats.md
+++ b/docs/en/interfaces/formats.md
@@ -31,8 +31,11 @@ The supported formats are:
| [JSON](#json) | ✗ | ✔ |
| [JSONAsString](#jsonasstring) | ✔ | ✗ |
| [JSONStrings](#jsonstrings) | ✗ | ✔ |
+| [JSONColumns](#jsoncolumns) | ✔ | ✔ |
+| [JSONColumnsWithMetadata](#jsoncolumnswithmetadata) | ✗ | ✔ |
| [JSONCompact](#jsoncompact) | ✗ | ✔ |
| [JSONCompactStrings](#jsoncompactstrings) | ✗ | ✔ |
+| [JSONCompactColumns](#jsoncompactcolumns) | ✔ | ✔ |
| [JSONEachRow](#jsoneachrow) | ✔ | ✔ |
| [JSONEachRowWithProgress](#jsoneachrowwithprogress) | ✗ | ✔ |
| [JSONStringsEachRow](#jsonstringseachrow) | ✔ | ✔ |
@@ -400,6 +403,8 @@ Both data output and parsing are supported in this format. For parsing, any orde
Parsing allows the presence of the additional field `tskv` without the equal sign or a value. This field is ignored.
+During import, columns with unknown names will be skipped if setting [input_format_skip_unknown_fields](../operations/settings/settings.md#settings-input-format-skip-unknown-fields) is set to 1.
+
## CSV {#csv}
Comma Separated Values format ([RFC](https://tools.ietf.org/html/rfc4180)).
@@ -459,15 +464,15 @@ SELECT SearchPhrase, count() AS c FROM test.hits GROUP BY SearchPhrase WITH TOTA
"meta":
[
{
- "name": "'hello'",
+ "name": "num",
+ "type": "Int32"
+ },
+ {
+ "name": "str",
"type": "String"
},
{
- "name": "multiply(42, number)",
- "type": "UInt64"
- },
- {
- "name": "range(5)",
+ "name": "arr",
"type": "Array(UInt8)"
}
],
@@ -475,25 +480,32 @@ SELECT SearchPhrase, count() AS c FROM test.hits GROUP BY SearchPhrase WITH TOTA
"data":
[
{
- "'hello'": "hello",
- "multiply(42, number)": "0",
- "range(5)": [0,1,2,3,4]
+ "num": 42,
+ "str": "hello",
+ "arr": [0,1]
},
{
- "'hello'": "hello",
- "multiply(42, number)": "42",
- "range(5)": [0,1,2,3,4]
+ "num": 43,
+ "str": "hello",
+ "arr": [0,1,2]
},
{
- "'hello'": "hello",
- "multiply(42, number)": "84",
- "range(5)": [0,1,2,3,4]
+ "num": 44,
+ "str": "hello",
+ "arr": [0,1,2,3]
}
],
"rows": 3,
- "rows_before_limit_at_least": 3
+ "rows_before_limit_at_least": 3,
+
+ "statistics":
+ {
+ "elapsed": 0.001137687,
+ "rows_read": 3,
+ "bytes_read": 24
+ }
}
```
@@ -528,15 +540,15 @@ Example:
"meta":
[
{
- "name": "'hello'",
+ "name": "num",
+ "type": "Int32"
+ },
+ {
+ "name": "str",
"type": "String"
},
{
- "name": "multiply(42, number)",
- "type": "UInt64"
- },
- {
- "name": "range(5)",
+ "name": "arr",
"type": "Array(UInt8)"
}
],
@@ -544,25 +556,95 @@ Example:
"data":
[
{
- "'hello'": "hello",
- "multiply(42, number)": "0",
- "range(5)": "[0,1,2,3,4]"
+ "num": "42",
+ "str": "hello",
+ "arr": "[0,1]"
},
{
- "'hello'": "hello",
- "multiply(42, number)": "42",
- "range(5)": "[0,1,2,3,4]"
+ "num": "43",
+ "str": "hello",
+ "arr": "[0,1,2]"
},
{
- "'hello'": "hello",
- "multiply(42, number)": "84",
- "range(5)": "[0,1,2,3,4]"
+ "num": "44",
+ "str": "hello",
+ "arr": "[0,1,2,3]"
}
],
"rows": 3,
- "rows_before_limit_at_least": 3
+ "rows_before_limit_at_least": 3,
+
+ "statistics":
+ {
+ "elapsed": 0.001403233,
+ "rows_read": 3,
+ "bytes_read": 24
+ }
+}
+```
+
+## JSONColumns {#jsoncolumns}
+
+In this format, all data is represented as a single JSON Object.
+Note that JSONColumns output format buffers all data in memory to output it as a single block and it can lead to high memory consumption.
+
+Example:
+```json
+{
+ "num": [42, 43, 44],
+ "str": ["hello", "hello", "hello"],
+ "arr": [[0,1], [0,1,2], [0,1,2,3]]
+}
+```
+
+During import, columns with unknown names will be skipped if setting [input_format_skip_unknown_fields](../operations/settings/settings.md#settings-input-format-skip-unknown-fields) is set to 1.
+Columns that are not present in the block will be filled with default values (you can use [input_format_defaults_for_omitted_fields](../operations/settings/settings.md#session_settings-input_format_defaults_for_omitted_fields) setting here)
+
+
+## JSONColumnsWithMetadata {#jsoncolumnsmonoblock}
+
+Differs from JSONColumns output format in that it also outputs some metadata and statistics (similar to JSON output format).
+This format buffers all data in memory and then outputs them as a single block, so, it can lead to high memory consumption.
+
+Example:
+```json
+{
+ "meta":
+ [
+ {
+ "name": "num",
+ "type": "Int32"
+ },
+ {
+ "name": "str",
+ "type": "String"
+ },
+
+ {
+ "name": "arr",
+ "type": "Array(UInt8)"
+ }
+ ],
+
+ "data":
+ {
+ "num": [42, 43, 44],
+ "str": ["hello", "hello", "hello"],
+ "arr": [[0,1], [0,1,2], [0,1,2,3]]
+ },
+
+ "rows": 3,
+
+ "rows_before_limit_at_least": 3,
+
+ "statistics":
+ {
+ "elapsed": 0.000272376,
+ "rows_read": 3,
+ "bytes_read": 24
+ }
}
```
@@ -618,71 +700,101 @@ Result:
Differs from JSON only in that data rows are output in arrays, not in objects.
+Examples:
+
+1) JSONCompact:
+```json
+{
+ "meta":
+ [
+ {
+ "name": "num",
+ "type": "Int32"
+ },
+ {
+ "name": "str",
+ "type": "String"
+ },
+ {
+ "name": "arr",
+ "type": "Array(UInt8)"
+ }
+ ],
+
+ "data":
+ [
+ [42, "hello", [0,1]],
+ [43, "hello", [0,1,2]],
+ [44, "hello", [0,1,2,3]]
+ ],
+
+ "rows": 3,
+
+ "rows_before_limit_at_least": 3,
+
+ "statistics":
+ {
+ "elapsed": 0.001222069,
+ "rows_read": 3,
+ "bytes_read": 24
+ }
+}
+```
+
+2) JSONCompactStrings
+```json
+{
+ "meta":
+ [
+ {
+ "name": "num",
+ "type": "Int32"
+ },
+ {
+ "name": "str",
+ "type": "String"
+ },
+ {
+ "name": "arr",
+ "type": "Array(UInt8)"
+ }
+ ],
+
+ "data":
+ [
+ ["42", "hello", "[0,1]"],
+ ["43", "hello", "[0,1,2]"],
+ ["44", "hello", "[0,1,2,3]"]
+ ],
+
+ "rows": 3,
+
+ "rows_before_limit_at_least": 3,
+
+ "statistics":
+ {
+ "elapsed": 0.001572097,
+ "rows_read": 3,
+ "bytes_read": 24
+ }
+}
+```
+
+## JSONCompactColumns {#jsoncompactcolumns}
+
+In this format, all data is represented as a single JSON Array.
+Note that JSONCompactColumns output format buffers all data in memory to output it as a single block and it can lead to high memory consumption
+
Example:
-
-```
-// JSONCompact
-{
- "meta":
- [
- {
- "name": "'hello'",
- "type": "String"
- },
- {
- "name": "multiply(42, number)",
- "type": "UInt64"
- },
- {
- "name": "range(5)",
- "type": "Array(UInt8)"
- }
- ],
-
- "data":
- [
- ["hello", "0", [0,1,2,3,4]],
- ["hello", "42", [0,1,2,3,4]],
- ["hello", "84", [0,1,2,3,4]]
- ],
-
- "rows": 3,
-
- "rows_before_limit_at_least": 3
-}
+```json
+[
+ [42, 43, 44],
+ ["hello", "hello", "hello"],
+ [[0,1], [0,1,2], [0,1,2,3]]
+]
```
-```
-// JSONCompactStrings
-{
- "meta":
- [
- {
- "name": "'hello'",
- "type": "String"
- },
- {
- "name": "multiply(42, number)",
- "type": "UInt64"
- },
- {
- "name": "range(5)",
- "type": "Array(UInt8)"
- }
- ],
-
- "data":
- [
- ["hello", "0", "[0,1,2,3,4]"],
- ["hello", "42", "[0,1,2,3,4]"],
- ["hello", "84", "[0,1,2,3,4]"]
- ],
-
- "rows": 3,
-
- "rows_before_limit_at_least": 3
-}
-```
+Columns that are not present in the block will be filled with default values (you can use [input_format_defaults_for_omitted_fields](../operations/settings/settings.md#session_settings-input_format_defaults_for_omitted_fields) setting here)
## JSONEachRow {#jsoneachrow}
## JSONStringsEachRow {#jsonstringseachrow}
@@ -699,15 +811,17 @@ When using these formats, ClickHouse outputs rows as separated, newline-delimite
When inserting the data, you should provide a separate JSON value for each row.
+In JSONEachRow/JSONStringsEachRow input formats columns with unknown names will be skipped if setting [input_format_skip_unknown_fields](../operations/settings/settings.md#settings-input-format-skip-unknown-fields) is set to 1.
+
## JSONEachRowWithProgress {#jsoneachrowwithprogress}
## JSONStringsEachRowWithProgress {#jsonstringseachrowwithprogress}
Differs from `JSONEachRow`/`JSONStringsEachRow` in that ClickHouse will also yield progress information as JSON values.
```json
-{"row":{"'hello'":"hello","multiply(42, number)":"0","range(5)":[0,1,2,3,4]}}
-{"row":{"'hello'":"hello","multiply(42, number)":"42","range(5)":[0,1,2,3,4]}}
-{"row":{"'hello'":"hello","multiply(42, number)":"84","range(5)":[0,1,2,3,4]}}
+{"row":{"num":42,"str":"hello","arr":[0,1]}}
+{"row":{"num":43,"str":"hello","arr":[0,1,2]}}
+{"row":{"num":44,"str":"hello","arr":[0,1,2,3]}}
{"progress":{"read_rows":"3","read_bytes":"24","written_rows":"0","written_bytes":"0","total_rows_to_read":"3"}}
```
@@ -728,11 +842,11 @@ Differs from `JSONCompactStringsEachRow` in that in that it also prints the head
Differs from `JSONCompactStringsEachRow` in that it also prints two header rows with column names and types, similar to [TabSeparatedWithNamesAndTypes](#tabseparatedwithnamesandtypes).
```json
-["'hello'", "multiply(42, number)", "range(5)"]
-["String", "UInt64", "Array(UInt8)"]
-["hello", "0", [0,1,2,3,4]]
-["hello", "42", [0,1,2,3,4]]
-["hello", "84", [0,1,2,3,4]]
+["num", "str", "arr"]
+["Int32", "String", "Array(UInt8)"]
+[42, "hello", [0,1]]
+[43, "hello", [0,1,2]]
+[44, "hello", [0,1,2,3]]
```
### Inserting Data {#inserting-data}
diff --git a/docs/en/operations/system-tables/tables.md b/docs/en/operations/system-tables/tables.md
index 8286d51aed6..6cf1490f14e 100644
--- a/docs/en/operations/system-tables/tables.md
+++ b/docs/en/operations/system-tables/tables.md
@@ -12,11 +12,13 @@ Columns:
- `name` ([String](../../sql-reference/data-types/string.md)) — Table name.
+- `uuid` ([UUID](../../sql-reference/data-types/uuid.md)) — Table uuid (Atomic database).
+
- `engine` ([String](../../sql-reference/data-types/string.md)) — Table engine name (without parameters).
- `is_temporary` ([UInt8](../../sql-reference/data-types/int-uint.md)) - Flag that indicates whether the table is temporary.
-- `data_path` ([String](../../sql-reference/data-types/string.md)) - Path to the table data in the file system.
+- `data_paths` ([Array](../../sql-reference/data-types/array.md)([String](../../sql-reference/data-types/string.md))) - Paths to the table data in the file systems.
- `metadata_path` ([String](../../sql-reference/data-types/string.md)) - Path to the table metadata in the file system.
@@ -60,6 +62,14 @@ Columns:
- `has_own_data` ([UInt8](../../sql-reference/data-types/int-uint.md)) — Flag that indicates whether the table itself stores some data on disk or only accesses some other source.
+- `loading_dependencies_database` ([Array](../../sql-reference/data-types/array.md)([String](../../sql-reference/data-types/string.md))) - Database loading dependencies (list of objects which should be loaded before the current object).
+
+- `loading_dependencies_table` ([Array](../../sql-reference/data-types/array.md)([String](../../sql-reference/data-types/string.md))) - Table loading dependencies (list of objects which should be loaded before the current object).
+
+- `loading_dependent_database` ([Array](../../sql-reference/data-types/array.md)([String](../../sql-reference/data-types/string.md))) - Dependent loading database.
+
+- `loading_dependent_table` ([Array](../../sql-reference/data-types/array.md)([String](../../sql-reference/data-types/string.md))) - Dependent loading table.
+
The `system.tables` table is used in `SHOW TABLES` query implementation.
**Example**
@@ -95,6 +105,10 @@ lifetime_rows: ᴺᵁᴸᴸ
lifetime_bytes: ᴺᵁᴸᴸ
comment:
has_own_data: 0
+loading_dependencies_database: []
+loading_dependencies_table: []
+loading_dependent_database: []
+loading_dependent_table: []
Row 2:
──────
@@ -122,4 +136,8 @@ lifetime_rows: ᴺᵁᴸᴸ
lifetime_bytes: ᴺᵁᴸᴸ
comment:
has_own_data: 0
+loading_dependencies_database: []
+loading_dependencies_table: []
+loading_dependent_database: []
+loading_dependent_table: []
```
diff --git a/docs/en/sql-reference/functions/random-functions.md b/docs/en/sql-reference/functions/random-functions.md
index 5e20a93da1f..3931898f081 100644
--- a/docs/en/sql-reference/functions/random-functions.md
+++ b/docs/en/sql-reference/functions/random-functions.md
@@ -96,10 +96,14 @@ SELECT fuzzBits(materialize('abacaba'), 0.1)
FROM numbers(3)
```
-\`\`\` text
-┌─fuzzBits(materialize(‘abacaba’), 0.1)─┐
-│ abaaaja │
-│ a\*cjab+ │
-│ aeca2A │
-└───────────────────────────────────────┘
+Result:
+
+``` text
+┌─fuzzBits(materialize('abacaba'), 0.1)─┐
+│ abaaaja │
+│ a*cjab+ │
+│ aeca2A │
+└───────────────────────────────────────┘
+```
+
diff --git a/docs/en/sql-reference/functions/uuid-functions.md b/docs/en/sql-reference/functions/uuid-functions.md
index d23b505a93f..08f281ba281 100644
--- a/docs/en/sql-reference/functions/uuid-functions.md
+++ b/docs/en/sql-reference/functions/uuid-functions.md
@@ -11,10 +11,16 @@ The functions for working with UUID are listed below.
Generates the [UUID](../data-types/uuid.md) of [version 4](https://tools.ietf.org/html/rfc4122#section-4.4).
+**Syntax**
+
``` sql
-generateUUIDv4()
+generateUUIDv4([x])
```
+**Arguments**
+
+- `x` — [Expression](../../sql-reference/syntax.md#syntax-expressions) resulting in any of the [supported data types](../../sql-reference/data-types/index.md#data_types). The resulting value is discarded, but the expression itself if used for bypassing [common subexpression elimination](../../sql-reference/functions/index.md#common-subexpression-elimination) if the function is called multiple times in one query. Optional parameter.
+
**Returned value**
The UUID type value.
@@ -37,6 +43,15 @@ SELECT * FROM t_uuid
└──────────────────────────────────────┘
```
+**Usage example if it is needed to generate multiple values in one row**
+
+```sql
+SELECT generateUUIDv4(1), generateUUIDv4(2)
+┌─generateUUIDv4(1)────────────────────┬─generateUUIDv4(2)────────────────────┐
+│ 2d49dc6e-ddce-4cd0-afb8-790956df54c1 │ 8abf8c13-7dea-4fdf-af3e-0e18767770e6 │
+└──────────────────────────────────────┴──────────────────────────────────────┘
+```
+
## empty {#empty}
Checks whether the input UUID is empty.
diff --git a/docs/en/sql-reference/functions/ym-dict-functions.md b/docs/en/sql-reference/functions/ym-dict-functions.md
index 85215957443..4fc727844e7 100644
--- a/docs/en/sql-reference/functions/ym-dict-functions.md
+++ b/docs/en/sql-reference/functions/ym-dict-functions.md
@@ -105,7 +105,7 @@ Example: `regionToCountry(toUInt32(213)) = 225` converts Moscow (213) to Russia
Converts a region to a continent. In every other way, this function is the same as ‘regionToCity’.
Example: `regionToContinent(toUInt32(213)) = 10001` converts Moscow (213) to Eurasia (10001).
-### regionToTopContinent (#regiontotopcontinent) {#regiontotopcontinent-regiontotopcontinent}
+### regionToTopContinent(id\[, geobase\]) {#regiontotopcontinentid-geobase}
Finds the highest continent in the hierarchy for the region.
diff --git a/docs/en/sql-reference/statements/create/user.md b/docs/en/sql-reference/statements/create/user.md
index 0aad0961a8b..34f0a13147c 100644
--- a/docs/en/sql-reference/statements/create/user.md
+++ b/docs/en/sql-reference/statements/create/user.md
@@ -29,12 +29,14 @@ There are multiple ways of user identification:
- `IDENTIFIED WITH no_password`
- `IDENTIFIED WITH plaintext_password BY 'qwerty'`
- `IDENTIFIED WITH sha256_password BY 'qwerty'` or `IDENTIFIED BY 'password'`
-- `IDENTIFIED WITH sha256_hash BY 'hash'`
+- `IDENTIFIED WITH sha256_hash BY 'hash'` or `IDENTIFIED WITH sha256_hash BY 'hash' SALT 'salt'`
- `IDENTIFIED WITH double_sha1_password BY 'qwerty'`
- `IDENTIFIED WITH double_sha1_hash BY 'hash'`
- `IDENTIFIED WITH ldap SERVER 'server_name'`
- `IDENTIFIED WITH kerberos` or `IDENTIFIED WITH kerberos REALM 'realm'`
+For identification with sha256_hash using `SALT` - hash must be calculated from concatination of 'password' and 'salt'.
+
## User Host {#user-host}
User host is a host from which a connection to ClickHouse server could be established. The host can be specified in the `HOST` query section in the following ways:
diff --git a/docs/en/sql-reference/statements/grant.md b/docs/en/sql-reference/statements/grant.md
index 1ee330061b5..b60114e10c5 100644
--- a/docs/en/sql-reference/statements/grant.md
+++ b/docs/en/sql-reference/statements/grant.md
@@ -170,6 +170,7 @@ Hierarchy of privileges:
- `SYSTEM FLUSH`
- `SYSTEM FLUSH DISTRIBUTED`
- `SYSTEM FLUSH LOGS`
+ - `CLUSTER` (see also `access_control_improvements.on_cluster_queries_require_cluster_grant` configuration directive)
- [INTROSPECTION](#grant-introspection)
- `addressToLine`
- `addressToLineWithInlines`
diff --git a/docs/ru/operations/system-tables/tables.md b/docs/ru/operations/system-tables/tables.md
index bf47051442e..ae5ca586a88 100644
--- a/docs/ru/operations/system-tables/tables.md
+++ b/docs/ru/operations/system-tables/tables.md
@@ -12,11 +12,13 @@
- `name` ([String](../../sql-reference/data-types/string.md)) — имя таблицы.
+- `uuid` ([UUID](../../sql-reference/data-types/uuid.md)) — Uuid таблицы (Atomic database).
+
- `engine` ([String](../../sql-reference/data-types/string.md)) — движок таблицы (без параметров).
- `is_temporary` ([UInt8](../../sql-reference/data-types/int-uint.md)) — флаг, указывающий на то, временная это таблица или нет.
-- `data_path` ([String](../../sql-reference/data-types/string.md)) — путь к данным таблицы в файловой системе.
+- `data_paths` ([Array](../../sql-reference/data-types/array.md)([String](../../sql-reference/data-types/string.md))) — пути к данным таблицы в файловых системах.
- `metadata_path` ([String](../../sql-reference/data-types/string.md)) — путь к табличным метаданным в файловой системе.
@@ -60,6 +62,14 @@
- `has_own_data` ([UInt8](../../sql-reference/data-types/int-uint.md)) — флаг, показывающий хранит ли таблица сама какие-то данные на диске или только обращается к какому-то другому источнику.
+- `loading_dependencies_database` ([Array](../../sql-reference/data-types/array.md)([String](../../sql-reference/data-types/string.md))) - базы данных необходимые для загрузки объекта.
+
+- `loading_dependencies_table` ([Array](../../sql-reference/data-types/array.md)([String](../../sql-reference/data-types/string.md))) - таблицы необходимые для загрузки объекта.
+
+- `loading_dependent_database` ([Array](../../sql-reference/data-types/array.md)([String](../../sql-reference/data-types/string.md))) - базы данных, которым объект необходим для загрузки.
+
+- `loading_dependent_table` ([Array](../../sql-reference/data-types/array.md)([String](../../sql-reference/data-types/string.md))) - таблицы, которым объект необходим для загрузки.
+
Таблица `system.tables` используется при выполнении запроса `SHOW TABLES`.
**Пример**
@@ -95,6 +105,10 @@ lifetime_rows: ᴺᵁᴸᴸ
lifetime_bytes: ᴺᵁᴸᴸ
comment:
has_own_data: 0
+loading_dependencies_database: []
+loading_dependencies_table: []
+loading_dependent_database: []
+loading_dependent_table: []
Row 2:
──────
@@ -122,4 +136,8 @@ lifetime_rows: ᴺᵁᴸᴸ
lifetime_bytes: ᴺᵁᴸᴸ
comment:
has_own_data: 0
+loading_dependencies_database: []
+loading_dependencies_table: []
+loading_dependent_database: []
+loading_dependent_table: []
```
diff --git a/docs/ru/sql-reference/functions/uuid-functions.md b/docs/ru/sql-reference/functions/uuid-functions.md
index babeb0d2693..554e78002b8 100644
--- a/docs/ru/sql-reference/functions/uuid-functions.md
+++ b/docs/ru/sql-reference/functions/uuid-functions.md
@@ -9,10 +9,16 @@ sidebar_label: "Функции для работы с UUID"
Генерирует идентификатор [UUID версии 4](https://tools.ietf.org/html/rfc4122#section-4.4).
+**Синтаксис**
+
``` sql
-generateUUIDv4()
+generateUUIDv4([x])
```
+**Аргументы**
+
+- `x` — [выражение](../syntax.md#syntax-expressions), возвращающее значение одного из [поддерживаемых типов данных](../data-types/index.md#data_types). Значение используется, чтобы избежать [склейки одинаковых выражений](index.md#common-subexpression-elimination), если функция вызывается несколько раз в одном запросе. Необязательный параметр.
+
**Возвращаемое значение**
Значение типа [UUID](../../sql-reference/functions/uuid-functions.md).
@@ -35,6 +41,15 @@ SELECT * FROM t_uuid
└──────────────────────────────────────┘
```
+**Пример использования, для генерации нескольких значений в одной строке**
+
+```sql
+SELECT generateUUIDv4(1), generateUUIDv4(2)
+┌─generateUUIDv4(1)────────────────────┬─generateUUIDv4(2)────────────────────┐
+│ 2d49dc6e-ddce-4cd0-afb8-790956df54c1 │ 8abf8c13-7dea-4fdf-af3e-0e18767770e6 │
+└──────────────────────────────────────┴──────────────────────────────────────┘
+```
+
## empty {#empty}
Проверяет, является ли входной UUID пустым.
diff --git a/docs/ru/sql-reference/statements/create/user.md b/docs/ru/sql-reference/statements/create/user.md
index 78c481e8eb7..d7da1748821 100644
--- a/docs/ru/sql-reference/statements/create/user.md
+++ b/docs/ru/sql-reference/statements/create/user.md
@@ -29,12 +29,14 @@ CREATE USER [IF NOT EXISTS | OR REPLACE] name1 [ON CLUSTER cluster_name1]
- `IDENTIFIED WITH no_password`
- `IDENTIFIED WITH plaintext_password BY 'qwerty'`
- `IDENTIFIED WITH sha256_password BY 'qwerty'` or `IDENTIFIED BY 'password'`
-- `IDENTIFIED WITH sha256_hash BY 'hash'`
+- `IDENTIFIED WITH sha256_hash BY 'hash'` or `IDENTIFIED WITH sha256_hash BY 'hash' SALT 'salt'`
- `IDENTIFIED WITH double_sha1_password BY 'qwerty'`
- `IDENTIFIED WITH double_sha1_hash BY 'hash'`
- `IDENTIFIED WITH ldap SERVER 'server_name'`
- `IDENTIFIED WITH kerberos` or `IDENTIFIED WITH kerberos REALM 'realm'`
+Для идентификации с sha256_hash используя `SALT` - хэш должен быть вычислен от конкатенации 'password' и 'salt'.
+
## Пользовательский хост
Пользовательский хост — это хост, с которого можно установить соединение с сервером ClickHouse. Хост задается в секции `HOST` следующими способами:
diff --git a/docs/tools/blog.py b/docs/tools/blog.py
deleted file mode 100644
index 9bb6beae972..00000000000
--- a/docs/tools/blog.py
+++ /dev/null
@@ -1,113 +0,0 @@
-#!/usr/bin/env python3
-import datetime
-import logging
-import os
-import time
-
-import nav # monkey patches mkdocs
-
-import mkdocs.commands
-from mkdocs import config
-from mkdocs import exceptions
-
-import mdx_clickhouse
-import redirects
-
-import util
-
-
-def build_for_lang(lang, args):
- logging.info(f"Building {lang} blog")
-
- try:
- theme_cfg = {
- "name": None,
- "custom_dir": os.path.join(os.path.dirname(__file__), "..", args.theme_dir),
- "language": lang,
- "direction": "ltr",
- "static_templates": ["404.html"],
- "extra": {
- "now": int(
- time.mktime(datetime.datetime.now().timetuple())
- ) # TODO better way to avoid caching
- },
- }
-
- # the following list of languages is sorted according to
- # https://en.wikipedia.org/wiki/List_of_languages_by_total_number_of_speakers
- languages = {"en": "English"}
-
- site_names = {"en": "ClickHouse Blog"}
-
- assert len(site_names) == len(languages)
-
- site_dir = os.path.join(args.blog_output_dir, lang)
-
- plugins = ["macros"]
- if args.htmlproofer:
- plugins.append("htmlproofer")
-
- website_url = "https://clickhouse.com"
- site_name = site_names.get(lang, site_names["en"])
- blog_nav, post_meta = nav.build_blog_nav(lang, args)
- raw_config = dict(
- site_name=site_name,
- site_url=f"{website_url}/blog/{lang}/",
- docs_dir=os.path.join(args.blog_dir, lang),
- site_dir=site_dir,
- strict=True,
- theme=theme_cfg,
- nav=blog_nav,
- copyright="©2016–2022 ClickHouse, Inc.",
- use_directory_urls=True,
- repo_name="ClickHouse/ClickHouse",
- repo_url="https://github.com/ClickHouse/ClickHouse/",
- edit_uri=f"edit/master/website/blog/{lang}",
- markdown_extensions=mdx_clickhouse.MARKDOWN_EXTENSIONS,
- plugins=plugins,
- extra=dict(
- now=datetime.datetime.now().isoformat(),
- rev=args.rev,
- rev_short=args.rev_short,
- rev_url=args.rev_url,
- website_url=website_url,
- events=args.events,
- languages=languages,
- includes_dir=os.path.join(os.path.dirname(__file__), "..", "_includes"),
- is_blog=True,
- post_meta=post_meta,
- today=datetime.date.today().isoformat(),
- ),
- )
-
- cfg = config.load_config(**raw_config)
- mkdocs.commands.build.build(cfg)
-
- redirects.build_blog_redirects(args)
-
- env = util.init_jinja2_env(args)
- with open(
- os.path.join(args.website_dir, "templates", "blog", "rss.xml"), "rb"
- ) as f:
- rss_template_string = f.read().decode("utf-8").strip()
- rss_template = env.from_string(rss_template_string)
- with open(os.path.join(args.blog_output_dir, lang, "rss.xml"), "w") as f:
- f.write(rss_template.render({"config": raw_config}))
-
- logging.info(f"Finished building {lang} blog")
-
- except exceptions.ConfigurationError as e:
- raise SystemExit("\n" + str(e))
-
-
-def build_blog(args):
- tasks = []
- for lang in args.blog_lang.split(","):
- if lang:
- tasks.append(
- (
- lang,
- args,
- )
- )
- util.run_function_in_parallel(build_for_lang, tasks, threads=False)
diff --git a/docs/tools/build.py b/docs/tools/build.py
index f084a8e5c0c..3756cf66794 100755
--- a/docs/tools/build.py
+++ b/docs/tools/build.py
@@ -1,144 +1,17 @@
#!/usr/bin/env python3
import argparse
-import datetime
import logging
import os
import shutil
import subprocess
import sys
-import time
-import jinja2
import livereload
-import markdown.util
-import nav # monkey patches mkdocs
-
-from mkdocs import config
-from mkdocs import exceptions
-import mkdocs.commands.build
-
-import blog
-import mdx_clickhouse
import redirects
-import util
import website
-from cmake_in_clickhouse_generator import generate_cmake_flags_files
-
-
-class ClickHouseMarkdown(markdown.extensions.Extension):
- class ClickHousePreprocessor(markdown.util.Processor):
- def run(self, lines):
- for line in lines:
- if "" not in line:
- yield line
-
- def extendMarkdown(self, md):
- md.preprocessors.register(
- self.ClickHousePreprocessor(), "clickhouse_preprocessor", 31
- )
-
-
-markdown.extensions.ClickHouseMarkdown = ClickHouseMarkdown
-
-
-def build_for_lang(lang, args):
- logging.info(f"Building {lang} docs")
-
- try:
- theme_cfg = {
- "name": None,
- "custom_dir": os.path.join(os.path.dirname(__file__), "..", args.theme_dir),
- "language": lang,
- "direction": "rtl" if lang == "fa" else "ltr",
- "static_templates": ["404.html"],
- "extra": {
- "now": int(
- time.mktime(datetime.datetime.now().timetuple())
- ) # TODO better way to avoid caching
- },
- }
-
- # the following list of languages is sorted according to
- # https://en.wikipedia.org/wiki/List_of_languages_by_total_number_of_speakers
- languages = {"en": "English", "zh": "中文", "ru": "Русский", "ja": "日本語"}
-
- site_names = {
- "en": "ClickHouse %s Documentation",
- "zh": "ClickHouse文档 %s",
- "ru": "Документация ClickHouse %s",
- "ja": "ClickHouseドキュメント %s",
- }
-
- assert len(site_names) == len(languages)
-
- site_dir = os.path.join(args.docs_output_dir, lang)
-
- plugins = ["macros"]
- if args.htmlproofer:
- plugins.append("htmlproofer")
-
- website_url = "https://clickhouse.com"
- site_name = site_names.get(lang, site_names["en"]) % ""
- site_name = site_name.replace(" ", " ")
-
- raw_config = dict(
- site_name=site_name,
- site_url=f"{website_url}/docs/{lang}/",
- docs_dir=os.path.join(args.docs_dir, lang),
- site_dir=site_dir,
- strict=True,
- theme=theme_cfg,
- copyright="©2016–2022 ClickHouse, Inc.",
- use_directory_urls=True,
- repo_name="ClickHouse/ClickHouse",
- repo_url="https://github.com/ClickHouse/ClickHouse/",
- edit_uri=f"edit/master/docs/{lang}",
- markdown_extensions=mdx_clickhouse.MARKDOWN_EXTENSIONS,
- plugins=plugins,
- extra=dict(
- now=datetime.datetime.now().isoformat(),
- rev=args.rev,
- rev_short=args.rev_short,
- rev_url=args.rev_url,
- website_url=website_url,
- events=args.events,
- languages=languages,
- includes_dir=os.path.join(os.path.dirname(__file__), "..", "_includes"),
- is_blog=False,
- ),
- )
-
- raw_config["nav"] = nav.build_docs_nav(lang, args)
-
- cfg = config.load_config(**raw_config)
-
- if not args.skip_multi_page:
- mkdocs.commands.build.build(cfg)
-
- mdx_clickhouse.PatchedMacrosPlugin.disabled = False
-
- logging.info(f"Finished building {lang} docs")
-
- except exceptions.ConfigurationError as e:
- raise SystemExit("\n" + str(e))
-
-
-def build_docs(args):
- tasks = []
- for lang in args.lang.split(","):
- if lang:
- tasks.append(
- (
- lang,
- args,
- )
- )
- util.run_function_in_parallel(build_for_lang, tasks, threads=False)
- redirects.build_docs_redirects(args)
-
def build(args):
if os.path.exists(args.output_dir):
@@ -147,14 +20,6 @@ def build(args):
if not args.skip_website:
website.build_website(args)
- if not args.skip_docs:
- generate_cmake_flags_files()
-
- build_docs(args)
-
- if not args.skip_blog:
- blog.build_blog(args)
-
if not args.skip_website:
website.process_benchmark_results(args)
website.minify_website(args)
@@ -171,20 +36,14 @@ if __name__ == "__main__":
arg_parser = argparse.ArgumentParser()
arg_parser.add_argument("--lang", default="en,ru,zh,ja")
- arg_parser.add_argument("--blog-lang", default="en")
- arg_parser.add_argument("--docs-dir", default=".")
arg_parser.add_argument("--theme-dir", default=website_dir)
arg_parser.add_argument("--website-dir", default=website_dir)
arg_parser.add_argument("--src-dir", default=src_dir)
- arg_parser.add_argument("--blog-dir", default=os.path.join(website_dir, "blog"))
arg_parser.add_argument("--output-dir", default="build")
arg_parser.add_argument("--nav-limit", type=int, default="0")
arg_parser.add_argument("--skip-multi-page", action="store_true")
arg_parser.add_argument("--skip-website", action="store_true")
- arg_parser.add_argument("--skip-blog", action="store_true")
- arg_parser.add_argument("--skip-docs", action="store_true")
arg_parser.add_argument("--htmlproofer", action="store_true")
- arg_parser.add_argument("--no-docs-macros", action="store_true")
arg_parser.add_argument("--livereload", type=int, default="0")
arg_parser.add_argument("--verbose", action="store_true")
@@ -196,11 +55,6 @@ if __name__ == "__main__":
logging.getLogger("MARKDOWN").setLevel(logging.INFO)
- args.docs_output_dir = os.path.join(os.path.abspath(args.output_dir), "docs")
- args.blog_output_dir = os.path.join(os.path.abspath(args.output_dir), "blog")
-
- from github import get_events
-
args.rev = (
subprocess.check_output("git rev-parse HEAD", shell=True)
.decode("utf-8")
@@ -212,9 +66,6 @@ if __name__ == "__main__":
.strip()
)
args.rev_url = f"https://github.com/ClickHouse/ClickHouse/commit/{args.rev}"
- args.events = get_events(args)
-
- from build import build
build(args)
@@ -223,9 +74,6 @@ if __name__ == "__main__":
new_args = sys.executable + " " + " ".join(new_args)
server = livereload.Server()
- server.watch(
- args.docs_dir + "**/*", livereload.shell(new_args, cwd="tools", shell=True)
- )
server.watch(
args.website_dir + "**/*",
livereload.shell(new_args, cwd="tools", shell=True),
diff --git a/docs/tools/cmake_in_clickhouse_generator.py b/docs/tools/cmake_in_clickhouse_generator.py
deleted file mode 100644
index 9bbc94fd206..00000000000
--- a/docs/tools/cmake_in_clickhouse_generator.py
+++ /dev/null
@@ -1,181 +0,0 @@
-import re
-import os
-from typing import TextIO, List, Tuple, Optional, Dict
-
-# name, default value, description
-Entity = Tuple[str, str, str]
-
-# https://regex101.com/r/R6iogw/12
-cmake_option_regex: str = (
- r"^\s*option\s*\(([A-Z_0-9${}]+)\s*(?:\"((?:.|\n)*?)\")?\s*(.*)?\).*$"
-)
-
-ch_master_url: str = "https://github.com/clickhouse/clickhouse/blob/master/"
-
-name_str: str = '[`{name}`](' + ch_master_url + "{path}#L{line})"
-default_anchor_str: str = "[`{name}`](#{anchor})"
-
-comment_var_regex: str = r"\${(.+)}"
-comment_var_replace: str = "`\\1`"
-
-table_header: str = """
-| Name | Default value | Description | Comment |
-|------|---------------|-------------|---------|
-"""
-
-# Needed to detect conditional variables (those which are defined twice)
-# name -> (path, values)
-entities: Dict[str, Tuple[str, str]] = {}
-
-
-def make_anchor(t: str) -> str:
- return "".join(
- ["-" if i == "_" else i.lower() for i in t if i.isalpha() or i == "_"]
- )
-
-
-def process_comment(comment: str) -> str:
- return re.sub(comment_var_regex, comment_var_replace, comment, flags=re.MULTILINE)
-
-
-def build_entity(path: str, entity: Entity, line_comment: Tuple[int, str]) -> None:
- (line, comment) = line_comment
- (name, description, default) = entity
-
- if name in entities:
- return
-
- if len(default) == 0:
- formatted_default: str = "`OFF`"
- elif default[0] == "$":
- formatted_default: str = "`{}`".format(default[2:-1])
- else:
- formatted_default: str = "`" + default + "`"
-
- formatted_name: str = name_str.format(
- anchor=make_anchor(name), name=name, path=path, line=line
- )
-
- formatted_description: str = "".join(description.split("\n"))
-
- formatted_comment: str = process_comment(comment)
-
- formatted_entity: str = "| {} | {} | {} | {} |".format(
- formatted_name, formatted_default, formatted_description, formatted_comment
- )
-
- entities[name] = path, formatted_entity
-
-
-def process_file(root_path: str, file_path: str, file_name: str) -> None:
- with open(os.path.join(file_path, file_name), "r") as cmake_file:
- contents: str = cmake_file.read()
-
- def get_line_and_comment(target: str) -> Tuple[int, str]:
- contents_list: List[str] = contents.split("\n")
- comment: str = ""
-
- for n, line in enumerate(contents_list):
- if "option" not in line.lower() or target not in line:
- continue
-
- for maybe_comment_line in contents_list[n - 1 :: -1]:
- if not re.match("\s*#\s*", maybe_comment_line):
- break
-
- comment = re.sub("\s*#\s*", "", maybe_comment_line) + " " + comment
-
- # line numbering starts with 1
- return n + 1, comment
-
- matches: Optional[List[Entity]] = re.findall(
- cmake_option_regex, contents, re.MULTILINE
- )
-
- file_rel_path_with_name: str = os.path.join(
- file_path[len(root_path) :], file_name
- )
- if file_rel_path_with_name.startswith("/"):
- file_rel_path_with_name = file_rel_path_with_name[1:]
-
- if matches:
- for entity in matches:
- build_entity(
- file_rel_path_with_name, entity, get_line_and_comment(entity[0])
- )
-
-
-def process_folder(root_path: str, name: str) -> None:
- for root, _, files in os.walk(os.path.join(root_path, name)):
- for f in files:
- if f == "CMakeLists.txt" or ".cmake" in f:
- process_file(root_path, root, f)
-
-
-def generate_cmake_flags_files() -> None:
- root_path: str = os.path.join(os.path.dirname(__file__), "..", "..")
-
- output_file_name: str = os.path.join(
- root_path, "docs/en/development/cmake-in-clickhouse.md"
- )
- header_file_name: str = os.path.join(
- root_path, "docs/_includes/cmake_in_clickhouse_header.md"
- )
- footer_file_name: str = os.path.join(
- root_path, "docs/_includes/cmake_in_clickhouse_footer.md"
- )
-
- process_file(root_path, root_path, "CMakeLists.txt")
- process_file(root_path, os.path.join(root_path, "programs"), "CMakeLists.txt")
-
- process_folder(root_path, "base")
- process_folder(root_path, "cmake")
- process_folder(root_path, "src")
-
- with open(output_file_name, "w") as f:
- with open(header_file_name, "r") as header:
- f.write(header.read())
-
- sorted_keys: List[str] = sorted(entities.keys())
- ignored_keys: List[str] = []
-
- f.write("### ClickHouse modes\n" + table_header)
-
- for k in sorted_keys:
- if k.startswith("ENABLE_CLICKHOUSE_"):
- f.write(entities[k][1] + "\n")
- ignored_keys.append(k)
-
- f.write(
- "\n### External libraries\nNote that ClickHouse uses forks of these libraries, see https://github.com/ClickHouse-Extras.\n"
- + table_header
- )
-
- for k in sorted_keys:
- if k.startswith("ENABLE_") and ".cmake" in entities[k][0]:
- f.write(entities[k][1] + "\n")
- ignored_keys.append(k)
-
- f.write("\n\n### Other flags\n" + table_header)
-
- for k in sorted(set(sorted_keys).difference(set(ignored_keys))):
- f.write(entities[k][1] + "\n")
-
- with open(footer_file_name, "r") as footer:
- f.write(footer.read())
-
- other_languages = [
- "docs/ja/development/cmake-in-clickhouse.md",
- "docs/zh/development/cmake-in-clickhouse.md",
- "docs/ru/development/cmake-in-clickhouse.md",
- ]
-
- for lang in other_languages:
- other_file_name = os.path.join(root_path, lang)
- if os.path.exists(other_file_name):
- os.unlink(other_file_name)
- os.symlink(output_file_name, other_file_name)
-
-
-if __name__ == "__main__":
- generate_cmake_flags_files()
diff --git a/docs/tools/easy_diff.py b/docs/tools/easy_diff.py
deleted file mode 100755
index 14e3ca91776..00000000000
--- a/docs/tools/easy_diff.py
+++ /dev/null
@@ -1,186 +0,0 @@
-#!/usr/bin/env python3
-# -*- coding: utf-8 -*-
-
-import os, sys
-import argparse
-import subprocess
-import contextlib
-from git import cmd
-from tempfile import NamedTemporaryFile
-
-SCRIPT_DESCRIPTION = """
- usage: ./easy_diff.py language/document path
-
- Show the difference between a language document and an English document.
-
- This script is based on the assumption that documents in other languages are fully synchronized with the en document at a commit.
-
- For example:
- Execute:
- ./easy_diff.py --no-pager zh/data_types
- Output:
- Need translate document:~/ClickHouse/docs/en/data_types/uuid.md
- Need link document:~/ClickHouse/docs/en/data_types/decimal.md to ~/ClickHouse/docs/zh/data_types/decimal.md
- diff --git a/docs/en/data_types/domains/ipv6.md b/docs/en/data_types/domains/ipv6.md
- index 1bfbe3400b..e2abaff017 100644
- --- a/docs/en/data_types/domains/ipv6.md
- +++ b/docs/en/data_types/domains/ipv6.md
- @@ -4,13 +4,13 @@
-
- ### Basic Usage
-
- -``` sql
- +```sql
- CREATE TABLE hits (url String, from IPv6) ENGINE = MergeTree() ORDER BY url;
-
- DESCRIBE TABLE hits;
- ```
-
- -```
- +```text
- ┌─name─┬─type───┬─default_type─┬─default_expression─┬─comment─┬─codec_expression─┐
- │ url │ String │ │ │ │ │
- │ from │ IPv6 │ │ │ │ │
- @@ -19,19 +19,19 @@ DESCRIBE TABLE hits;
-
- OR you can use `IPv6` domain as a key:
-
- -``` sql
- +```sql
- CREATE TABLE hits (url String, from IPv6) ENGINE = MergeTree() ORDER BY from;
- ... MORE
-
- OPTIONS:
- -h, --help show this help message and exit
- --no-pager use stdout as difference result output
-"""
-
-SCRIPT_PATH = os.path.abspath(__file__)
-CLICKHOUSE_REPO_HOME = os.path.join(os.path.dirname(SCRIPT_PATH), "..", "..")
-SCRIPT_COMMAND_EXECUTOR = cmd.Git(CLICKHOUSE_REPO_HOME)
-
-SCRIPT_COMMAND_PARSER = argparse.ArgumentParser(add_help=False)
-SCRIPT_COMMAND_PARSER.add_argument("path", type=bytes, nargs="?", default=None)
-SCRIPT_COMMAND_PARSER.add_argument("--no-pager", action="store_true", default=False)
-SCRIPT_COMMAND_PARSER.add_argument("-h", "--help", action="store_true", default=False)
-
-
-def execute(commands):
- return SCRIPT_COMMAND_EXECUTOR.execute(commands)
-
-
-def get_hash(file_name):
- return execute(["git", "log", "-n", "1", '--pretty=format:"%H"', file_name])
-
-
-def diff_file(reference_file, working_file, out):
- if not os.path.exists(reference_file):
- raise RuntimeError(
- "reference file [" + os.path.abspath(reference_file) + "] is not exists."
- )
-
- if os.path.islink(working_file):
- out.writelines(["Need translate document:" + os.path.abspath(reference_file)])
- elif not os.path.exists(working_file):
- out.writelines(
- [
- "Need link document "
- + os.path.abspath(reference_file)
- + " to "
- + os.path.abspath(working_file)
- ]
- )
- elif get_hash(working_file) != get_hash(reference_file):
- out.writelines(
- [
- (
- execute(
- [
- "git",
- "diff",
- get_hash(working_file).strip('"'),
- reference_file,
- ]
- ).encode("utf-8")
- )
- ]
- )
-
- return 0
-
-
-def diff_directory(reference_directory, working_directory, out):
- if not os.path.isdir(reference_directory):
- return diff_file(reference_directory, working_directory, out)
-
- for list_item in os.listdir(reference_directory):
- working_item = os.path.join(working_directory, list_item)
- reference_item = os.path.join(reference_directory, list_item)
- if (
- diff_file(reference_item, working_item, out)
- if os.path.isfile(reference_item)
- else diff_directory(reference_item, working_item, out) != 0
- ):
- return 1
-
- return 0
-
-
-def find_language_doc(custom_document, other_language="en", children=[]):
- if len(custom_document) == 0:
- raise RuntimeError(
- "The "
- + os.path.join(custom_document, *children)
- + " is not in docs directory."
- )
-
- if os.path.samefile(os.path.join(CLICKHOUSE_REPO_HOME, "docs"), custom_document):
- return os.path.join(CLICKHOUSE_REPO_HOME, "docs", other_language, *children[1:])
- children.insert(0, os.path.split(custom_document)[1])
- return find_language_doc(
- os.path.split(custom_document)[0], other_language, children
- )
-
-
-class ToPager:
- def __init__(self, temp_named_file):
- self.temp_named_file = temp_named_file
-
- def writelines(self, lines):
- self.temp_named_file.writelines(lines)
-
- def close(self):
- self.temp_named_file.flush()
- git_pager = execute(["git", "var", "GIT_PAGER"])
- subprocess.check_call([git_pager, self.temp_named_file.name])
- self.temp_named_file.close()
-
-
-class ToStdOut:
- def writelines(self, lines):
- self.system_stdout_stream.writelines(lines)
-
- def close(self):
- self.system_stdout_stream.flush()
-
- def __init__(self, system_stdout_stream):
- self.system_stdout_stream = system_stdout_stream
-
-
-if __name__ == "__main__":
- arguments = SCRIPT_COMMAND_PARSER.parse_args()
- if arguments.help or not arguments.path:
- sys.stdout.write(SCRIPT_DESCRIPTION)
- sys.exit(0)
-
- working_language = os.path.join(CLICKHOUSE_REPO_HOME, "docs", arguments.path)
- with contextlib.closing(
- ToStdOut(sys.stdout)
- if arguments.no_pager
- else ToPager(NamedTemporaryFile("r+"))
- ) as writer:
- exit(
- diff_directory(
- find_language_doc(working_language), working_language, writer
- )
- )
diff --git a/docs/tools/github.py b/docs/tools/github.py
deleted file mode 100644
index 3a6f155e25d..00000000000
--- a/docs/tools/github.py
+++ /dev/null
@@ -1,41 +0,0 @@
-import collections
-import copy
-import io
-import logging
-import os
-import random
-import sys
-import tarfile
-import time
-
-import requests
-
-import util
-
-
-def get_events(args):
- events = []
- skip = True
- with open(os.path.join(args.docs_dir, "..", "README.md")) as f:
- for line in f:
- if skip:
- if "Upcoming Events" in line:
- skip = False
- else:
- if not line:
- continue
- line = line.strip().split("](")
- if len(line) == 2:
- tail = line[1].split(") ")
- events.append(
- {
- "signup_link": tail[0],
- "event_name": line[0].replace("* [", ""),
- "event_date": tail[1].replace("on ", "").replace(".", ""),
- }
- )
- return events
-
-
-if __name__ == "__main__":
- logging.basicConfig(level=logging.DEBUG, stream=sys.stderr)
diff --git a/docs/tools/nav.py b/docs/tools/nav.py
deleted file mode 100644
index e3df85bbe4e..00000000000
--- a/docs/tools/nav.py
+++ /dev/null
@@ -1,190 +0,0 @@
-import collections
-import datetime
-import hashlib
-import logging
-import os
-
-import mkdocs.structure.nav
-
-import util
-
-
-def find_first_header(content):
- for line in content.split("\n"):
- if line.startswith("#"):
- no_hash = line.lstrip("#")
- return no_hash.split("{", 1)[0].strip()
-
-
-def build_nav_entry(root, args):
- if root.endswith("images"):
- return None, None, None
- result_items = []
- index_meta, index_content = util.read_md_file(os.path.join(root, "index.md"))
- current_title = index_meta.get("toc_folder_title", index_meta.get("toc_title"))
- current_title = current_title or index_meta.get(
- "title", find_first_header(index_content)
- )
- for filename in os.listdir(root):
- path = os.path.join(root, filename)
- if os.path.isdir(path):
- prio, title, payload = build_nav_entry(path, args)
- if title and payload:
- result_items.append((prio, title, payload))
- elif filename.endswith(".md"):
- path = os.path.join(root, filename)
-
- meta = ""
- content = ""
-
- try:
- meta, content = util.read_md_file(path)
- except:
- print("Error in file: {}".format(path))
- raise
-
- path = path.split("/", 2)[-1]
- title = meta.get("toc_title", find_first_header(content))
- if title:
- title = title.strip().rstrip(".")
- else:
- title = meta.get("toc_folder_title", "hidden")
- prio = meta.get("toc_priority", 9999)
- logging.debug(f"Nav entry: {prio}, {title}, {path}")
- if meta.get("toc_hidden") or not content.strip():
- title = "hidden"
- if title == "hidden":
- title = "hidden-" + hashlib.sha1(content.encode("utf-8")).hexdigest()
- if args.nav_limit and len(result_items) >= args.nav_limit:
- break
- result_items.append((prio, title, path))
- result_items = sorted(result_items, key=lambda x: (x[0], x[1]))
- result = collections.OrderedDict([(item[1], item[2]) for item in result_items])
- if index_meta.get("toc_hidden_folder"):
- current_title += "|hidden-folder"
- return index_meta.get("toc_priority", 10000), current_title, result
-
-
-def build_docs_nav(lang, args):
- docs_dir = os.path.join(args.docs_dir, lang)
- _, _, nav = build_nav_entry(docs_dir, args)
- result = []
- index_key = None
- for key, value in list(nav.items()):
- if key and value:
- if value == "index.md":
- index_key = key
- continue
- result.append({key: value})
- if args.nav_limit and len(result) >= args.nav_limit:
- break
- if index_key:
- key = list(result[0].keys())[0]
- result[0][key][index_key] = "index.md"
- result[0][key].move_to_end(index_key, last=False)
- return result
-
-
-def build_blog_nav(lang, args):
- blog_dir = os.path.join(args.blog_dir, lang)
- years = sorted(os.listdir(blog_dir), reverse=True)
- result_nav = [{"hidden": "index.md"}]
- post_meta = collections.OrderedDict()
- for year in years:
- year_dir = os.path.join(blog_dir, year)
- if not os.path.isdir(year_dir):
- continue
- result_nav.append({year: collections.OrderedDict()})
- posts = []
- post_meta_items = []
- for post in os.listdir(year_dir):
- post_path = os.path.join(year_dir, post)
- if not post.endswith(".md"):
- raise RuntimeError(
- f"Unexpected non-md file in posts folder: {post_path}"
- )
- meta, _ = util.read_md_file(post_path)
- post_date = meta["date"]
- post_title = meta["title"]
- if datetime.date.fromisoformat(post_date) > datetime.date.today():
- continue
- posts.append(
- (
- post_date,
- post_title,
- os.path.join(year, post),
- )
- )
- if post_title in post_meta:
- raise RuntimeError(f"Duplicate post title: {post_title}")
- if not post_date.startswith(f"{year}-"):
- raise RuntimeError(
- f"Post date {post_date} doesn't match the folder year {year}: {post_title}"
- )
- post_url_part = post.replace(".md", "")
- post_meta_items.append(
- (
- post_date,
- {
- "date": post_date,
- "title": post_title,
- "image": meta.get("image"),
- "url": f"/blog/{lang}/{year}/{post_url_part}/",
- },
- )
- )
- for _, title, path in sorted(posts, reverse=True):
- result_nav[-1][year][title] = path
- for _, post_meta_item in sorted(
- post_meta_items, reverse=True, key=lambda item: item[0]
- ):
- post_meta[post_meta_item["title"]] = post_meta_item
- return result_nav, post_meta
-
-
-def _custom_get_navigation(files, config):
- nav_config = config["nav"] or mkdocs.structure.nav.nest_paths(
- f.src_path for f in files.documentation_pages()
- )
- items = mkdocs.structure.nav._data_to_navigation(nav_config, files, config)
- if not isinstance(items, list):
- items = [items]
-
- pages = mkdocs.structure.nav._get_by_type(items, mkdocs.structure.nav.Page)
-
- mkdocs.structure.nav._add_previous_and_next_links(pages)
- mkdocs.structure.nav._add_parent_links(items)
-
- missing_from_config = [
- file for file in files.documentation_pages() if file.page is None
- ]
- if missing_from_config:
- files._files = [
- file for file in files._files if file not in missing_from_config
- ]
-
- links = mkdocs.structure.nav._get_by_type(items, mkdocs.structure.nav.Link)
- for link in links:
- scheme, netloc, path, params, query, fragment = mkdocs.structure.nav.urlparse(
- link.url
- )
- if scheme or netloc:
- mkdocs.structure.nav.log.debug(
- "An external link to '{}' is included in "
- "the 'nav' configuration.".format(link.url)
- )
- elif link.url.startswith("/"):
- mkdocs.structure.nav.log.debug(
- "An absolute path to '{}' is included in the 'nav' configuration, "
- "which presumably points to an external resource.".format(link.url)
- )
- else:
- msg = (
- "A relative path to '{}' is included in the 'nav' configuration, "
- "which is not found in the documentation files".format(link.url)
- )
- mkdocs.structure.nav.log.warning(msg)
- return mkdocs.structure.nav.Navigation(items, pages)
-
-
-mkdocs.structure.nav.get_navigation = _custom_get_navigation
diff --git a/docs/tools/redirects.py b/docs/tools/redirects.py
index 5d222376683..1b5490a040f 100644
--- a/docs/tools/redirects.py
+++ b/docs/tools/redirects.py
@@ -27,45 +27,6 @@ def write_redirect_html(out_path, to_url):
)
-def build_redirect_html(args, base_prefix, lang, output_dir, from_path, to_path):
- out_path = os.path.join(
- output_dir,
- lang,
- from_path.replace("/index.md", "/index.html").replace(".md", "/index.html"),
- )
- target_path = to_path.replace("/index.md", "/").replace(".md", "/")
-
- if target_path[0:7] != "http://" and target_path[0:8] != "https://":
- to_url = f"/{base_prefix}/{lang}/{target_path}"
- else:
- to_url = target_path
-
- to_url = to_url.strip()
- write_redirect_html(out_path, to_url)
-
-
-def build_docs_redirects(args):
- with open(os.path.join(args.docs_dir, "redirects.txt"), "r") as f:
- for line in f:
- for lang in args.lang.split(","):
- from_path, to_path = line.split(" ", 1)
- build_redirect_html(
- args, "docs", lang, args.docs_output_dir, from_path, to_path
- )
-
-
-def build_blog_redirects(args):
- for lang in args.blog_lang.split(","):
- redirects_path = os.path.join(args.blog_dir, lang, "redirects.txt")
- if os.path.exists(redirects_path):
- with open(redirects_path, "r") as f:
- for line in f:
- from_path, to_path = line.split(" ", 1)
- build_redirect_html(
- args, "blog", lang, args.blog_output_dir, from_path, to_path
- )
-
-
def build_static_redirects(args):
for static_redirect in [
("benchmark.html", "/benchmark/dbms/"),
diff --git a/docs/tools/requirements.txt b/docs/tools/requirements.txt
index dd641c13629..b6f2d4549e5 100644
--- a/docs/tools/requirements.txt
+++ b/docs/tools/requirements.txt
@@ -1,39 +1,32 @@
Babel==2.9.1
-backports-abc==0.5
-backports.functools-lru-cache==1.6.1
-beautifulsoup4==4.9.1
-certifi==2020.4.5.2
-chardet==3.0.4
-click==7.1.2
-closure==20191111
-cssmin==0.2.0
-future==0.18.2
-htmlmin==0.1.12
-idna==2.10
Jinja2==3.0.3
-jinja2-highlight==0.6.1
-jsmin==3.0.0
-livereload==2.6.3
Markdown==3.3.2
-MarkupSafe==2.1.0
-mkdocs==1.3.0
-mkdocs-htmlproofer-plugin==0.0.3
-mkdocs-macros-plugin==0.4.20
-nltk==3.7
-nose==1.3.7
-protobuf==3.14.0
-numpy==1.21.2
-pymdown-extensions==8.0
-python-slugify==4.0.1
+MarkupSafe==2.1.1
+MarkupSafe==2.1.1
PyYAML==6.0
-repackage==0.7.3
-requests==2.25.1
-singledispatch==3.4.0.3
+Pygments>=2.12.0
+beautifulsoup4==4.9.1
+click==7.1.2
+ghp_import==2.1.0
+importlib_metadata==4.11.4
+jinja2-highlight==0.6.1
+livereload==2.6.3
+mergedeep==1.3.4
+mkdocs-macros-plugin==0.4.20
+mkdocs-macros-test==0.1.0
+mkdocs-material==8.2.15
+mkdocs==1.3.0
+mkdocs_material_extensions==1.0.3
+packaging==21.3
+pygments==2.12.0
+pymdown_extensions==9.4
+pyparsing==3.0.9
+python-slugify==4.0.1
+python_dateutil==2.8.2
+pytz==2022.1
six==1.15.0
-soupsieve==2.0.1
+soupsieve==2.3.2
termcolor==1.1.0
+text_unidecode==1.3
tornado==6.1
-Unidecode==1.1.1
-urllib3>=1.26.8
-Pygments>=2.11.2
-
+zipp==3.8.0
diff --git a/docs/tools/util.py b/docs/tools/util.py
index ec670725122..a5ebb1b11b2 100644
--- a/docs/tools/util.py
+++ b/docs/tools/util.py
@@ -124,7 +124,7 @@ def init_jinja2_env(args):
env = jinja2.Environment(
loader=jinja2.FileSystemLoader(
- [args.website_dir, os.path.join(args.docs_dir, "_includes")]
+ [args.website_dir, os.path.join(args.src_dir, "docs", "_includes")]
),
extensions=["jinja2.ext.i18n", "jinja2_highlight.HighlightExtension"],
)
diff --git a/docs/tools/webpack.config.js b/docs/tools/webpack.config.js
deleted file mode 100644
index e0dea964101..00000000000
--- a/docs/tools/webpack.config.js
+++ /dev/null
@@ -1,81 +0,0 @@
-const path = require('path');
-const jsPath = path.resolve(__dirname, '../../website/src/js');
-const scssPath = path.resolve(__dirname, '../../website/src/scss');
-
-console.log(path.resolve(__dirname, 'node_modules/bootstrap', require('bootstrap/package.json').sass));
-
-module.exports = {
-
- mode: ('development' === process.env.NODE_ENV) && 'development' || 'production',
-
- ...(('development' === process.env.NODE_ENV) && {
- watch: true,
- }),
-
- entry: [
- path.resolve(scssPath, 'bootstrap.scss'),
- path.resolve(scssPath, 'main.scss'),
- path.resolve(jsPath, 'main.js'),
- ],
-
- output: {
- path: path.resolve(__dirname, '../../website'),
- filename: 'js/main.js',
- },
-
- resolve: {
- alias: {
- bootstrap: path.resolve(__dirname, 'node_modules/bootstrap', require('bootstrap/package.json').sass),
- },
- },
-
- module: {
- rules: [{
- test: /\.js$/,
- exclude: /(node_modules)/,
- use: [{
- loader: 'babel-loader',
- options: {
- presets: ['@babel/preset-env'],
- },
- }],
- }, {
- test: /\.scss$/,
- use: [{
- loader: 'file-loader',
- options: {
- sourceMap: true,
- outputPath: (url, entryPath, context) => {
- if (0 === entryPath.indexOf(scssPath)) {
- const outputFile = entryPath.slice(entryPath.lastIndexOf('/') + 1, -5)
- const outputPath = entryPath.slice(0, entryPath.lastIndexOf('/')).slice(scssPath.length + 1)
- return `./css/${outputPath}/${outputFile}.css`
- }
- return `./css/${url}`
- },
- },
- }, {
- loader: 'postcss-loader',
- options: {
- options: {},
- plugins: () => ([
- require('autoprefixer'),
- ('production' === process.env.NODE_ENV) && require('cssnano'),
- ].filter(plugin => plugin)),
- }
- }, {
- loader: 'sass-loader',
- options: {
- implementation: require('sass'),
- implementation: require('sass'),
- sourceMap: ('development' === process.env.NODE_ENV),
- sassOptions: {
- importer: require('node-sass-glob-importer')(),
- precision: 10,
- },
- },
- }],
- }],
- },
-
-};
diff --git a/docs/zh/sql-reference/functions/encoding-functions.md b/docs/zh/sql-reference/functions/encoding-functions.md
index f1152965d2d..b9a3cbf0550 100644
--- a/docs/zh/sql-reference/functions/encoding-functions.md
+++ b/docs/zh/sql-reference/functions/encoding-functions.md
@@ -68,12 +68,306 @@ SELECT char(0xE4, 0xBD, 0xA0, 0xE5, 0xA5, 0xBD) AS hello;
## hex {#hex}
-接受`String`，`unsigned integer`，`Date`或`DateTime`类型的参数。返回包含参数的十六进制表示的字符串。使用大写字母`A-F`。不使用`0x`前缀或`h`后缀。对于字符串，所有字节都简单地编码为两个十六进制数字。数字转换为大端（«易阅读»）格式。对于数字，去除其中较旧的零，但仅限整个字节。例如，`hex（1）='01'`。 `Date`被编码为自Unix时间开始以来的天数。 `DateTime`编码为自Unix时间开始以来的秒数。
+返回包含参数的十六进制表示的字符串。
-## unhex(str) {#unhexstr}
+别名为： `HEX`。
-接受包含任意数量的十六进制数字的字符串，并返回包含相应字节的字符串。支持大写和小写字母A-F。十六进制数字的数量不必是偶数。如果是奇数，则最后一位数被解释为00-0F字节的低位。如果参数字符串包含除十六进制数字以外的任何内容，则返回一些实现定义的结果（不抛出异常）。
-如果要将结果转换为数字，可以使用«reverse»和«reinterpretAsType»函数。
+**语法**
+
+``` sql
+hex(arg)
+```
+
+该函数使用大写字母`A-F`，不使用任何前缀（如`0x`）或后缀（如`h`）
+
+对于整数参数，它从高到低（大端或“人类可读”顺序）打印十六进制数字（“半字节”）。它从左侧第一个非零字节开始（省略前导零字节），但即使前导数字为零，也始终打印每个字节的两个数字。
+
+类型为[Date](../../sql-reference/data-types/date.md)和[DateTime](../../sql-reference/data-types/datetime.md)的值将被格式化为相应的整数（日期为 Epoch 以来的天数，DateTime 为 Unix Timestamp 的值）。
+
+对于[String](../../sql-reference/data-types/string.md)和[FixedString](../../sql-reference/data-types/fixedstring.md)，所有字节都被简单地编码为两个十六进制数字。零字节不会被省略。
+
+类型为[Float](../../sql-reference/data-types/float.md)和[Decimal](../../sql-reference/data-types/decimal.md)的值被编码为它们在内存中的表示。由于我们支持小端架构，它们以小端编码。零前导尾随字节不会被省略。
+
+类型为[UUID](../data-types/uuid.md)的值被编码为大端顺序字符串。
+
+**参数**
+
+- `arg` — 要转换为十六进制的值。类型为[String](../../sql-reference/data-types/string.md)，[UInt](../../sql-reference/data-types/int-uint.md)，[Float](../../sql-reference/data-types/float.md)，[Decimal](../../sql-reference/data-types/decimal.md)，[Date](../../sql-reference/data-types/date.md)或者[DateTime](../../sql-reference/data-types/datetime.md)。
+
+**返回值**
+
+- 具有参数的十六进制表示的字符串。
+
+类型为：[String](../../sql-reference/data-types/string.md)。
+
+**示例**
+
+查询语句：
+
+``` sql
+SELECT hex(1);
+```
+
+结果：
+
+``` text
+01
+```
+
+查询语句：
+
+``` sql
+SELECT hex(toFloat32(number)) AS hex_presentation FROM numbers(15, 2);
+```
+
+结果：
+
+``` text
+┌─hex_presentation─┐
+│ 00007041 │
+│ 00008041 │
+└──────────────────┘
+```
+
+查询语句：
+
+``` sql
+SELECT hex(toFloat64(number)) AS hex_presentation FROM numbers(15, 2);
+```
+
+结果：
+
+``` text
+┌─hex_presentation─┐
+│ 0000000000002E40 │
+│ 0000000000003040 │
+└──────────────────┘
+```
+
+查询语句：
+
+``` sql
+SELECT lower(hex(toUUID('61f0c404-5cb3-11e7-907b-a6006ad3dba0'))) as uuid_hex
+```
+
+结果：
+
+``` text
+┌─uuid_hex─────────────────────────┐
+│ 61f0c4045cb311e7907ba6006ad3dba0 │
+└──────────────────────────────────┘
+```
+
+## unhex {#unhexstr}
+
+执行[hex](#hex)函数的相反操作。它将每对十六进制数字（在参数中）解释为一个数字，并将其转换为该数字表示的字节。返回值是一个二进制字符串 (BLOB)。
+
+如果要将结果转换为数字，可以使用 [reverse](../../sql-reference/functions/string-functions.md#reverse) 和 [reinterpretAs<Type>](../../sql-reference/functions/type-conversion-functions.md#type-conversion-functions) 函数。
+
+:::注意
+如果从 `clickhouse-client` 中调用 `unhex`，二进制字符串将使用 UTF-8 显示。
+:::
+
+别名为：`UNHEX`。
+
+**语法**
+
+``` sql
+unhex(arg)
+```
+
+**参数**
+
+- `arg` — 包含任意数量的十六进制数字的字符串。类型为：[String](../../sql-reference/data-types/string.md)。
+
+支持大写和小写字母A-F。十六进制数字的数量不必是偶数。如果是奇数，则最后一位数被解释为00-0F字节的低位。如果参数字符串包含除十六进制数字以外的任何内容，则返回一些实现定义的结果（不抛出异常）。对于数字参数， unhex()不执行 hex(N) 的倒数。
+
+**返回值**
+
+- 二进制字符串 (BLOB)。
+
+类型为： [String](../../sql-reference/data-types/string.md)。
+
+**示例**
+
+查询语句：
+``` sql
+SELECT unhex('303132'), UNHEX('4D7953514C');
+```
+
+结果：
+``` text
+┌─unhex('303132')─┬─unhex('4D7953514C')─┐
+│ 012 │ MySQL │
+└─────────────────┴─────────────────────┘
+```
+
+查询语句：
+
+``` sql
+SELECT reinterpretAsUInt64(reverse(unhex('FFF'))) AS num;
+```
+
+结果：
+
+``` text
+┌──num─┐
+│ 4095 │
+└──────┘
+```
+
+## bin {#bin}
+
+返回一个包含参数二进制表示的字符串。
+
+**语法**
+
+``` sql
+bin(arg)
+```
+
+别名为： `BIN`。
+
+对于整数参数，它从最高有效到最低有效（大端或“人类可读”顺序）打印 bin 数字。它从最重要的非零字节开始（省略前导零字节），但如果前导数字为零，则始终打印每个字节的八位数字。
+
+类型为[Date](../../sql-reference/data-types/date.md)和[DateTime](../../sql-reference/data-types/datetime.md)的值被格式化为相应的整数（`Date` 为 Epoch 以来的天数，`DateTime` 为 Unix Timestamp 的值）。
+
+对于[String](../../sql-reference/data-types/string.md)和[FixedString](../../sql-reference/data-types/fixedstring.md)，所有字节都被简单地编码为八个二进制数。零字节不会被省略。
+
+类型为[Float](../../sql-reference/data-types/float.md)和[Decimal](../../sql-reference/data-types/decimal.md)的值被编码为它们在内存中的表示。由于我们支持小端架构，它们以小端编码。零前导尾随字节不会被省略。
+
+类型为[UUID](../data-types/uuid.md)的值被编码为大端顺序字符串。
+
+**参数**
+
+- `arg` — 要转换为二进制的值。类型为[String](../../sql-reference/data-types/string.md)，[FixedString](../../sql-reference/data-types/fixedstring.md)，[UInt](../../sql-reference/data-types/int-uint.md)，[Float](../../sql-reference/data-types/float.md)，[Decimal](../../sql-reference/data-types/decimal.md)，[Date](../../sql-reference/data-types/date.md)或者[DateTime](../../sql-reference/data-types/datetime.md)。
+
+**返回值**
+
+- 具有参数的二进制表示的字符串。
+
+类型为： [String](../../sql-reference/data-types/string.md)。
+
+**示例**
+
+查询语句：
+
+``` sql
+SELECT bin(14);
+```
+
+结果：
+
+``` text
+┌─bin(14)──┐
+│ 00001110 │
+└──────────┘
+```
+
+查询语句：
+
+``` sql
+SELECT bin(toFloat32(number)) AS bin_presentation FROM numbers(15, 2);
+```
+
+结果：
+
+``` text
+┌─bin_presentation─────────────────┐
+│ 00000000000000000111000001000001 │
+│ 00000000000000001000000001000001 │
+└──────────────────────────────────┘
+```
+
+查询语句：
+
+``` sql
+SELECT bin(toFloat64(number)) AS bin_presentation FROM numbers(15, 2);
+```
+
+结果：
+
+``` text
+┌─bin_presentation─────────────────────────────────────────────────┐
+│ 0000000000000000000000000000000000000000000000000010111001000000 │
+│ 0000000000000000000000000000000000000000000000000011000001000000 │
+└──────────────────────────────────────────────────────────────────┘
+```
+
+查询语句：
+
+``` sql
+SELECT bin(toUUID('61f0c404-5cb3-11e7-907b-a6006ad3dba0')) as bin_uuid
+```
+
+结果：
+
+``` text
+┌─bin_uuid─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
+│ 01100001111100001100010000000100010111001011001100010001111001111001000001111011101001100000000001101010110100111101101110100000 │
+└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
+```
+
+
+## unbin {#unbinstr}
+
+将每对二进制数字（在参数中）解释为一个数字，并将其转换为该数字表示的字节。这些函数执行与 [bin](#bin) 相反的操作。
+
+**语法**
+
+``` sql
+unbin(arg)
+```
+
+别名为： `UNBIN`。
+
+对于数字参数，`unbin()` 不会返回 `bin()` 的倒数。如果要将结果转换为数字，可以使用[reverse](../../sql-reference/functions/string-functions.md#reverse) 和 [reinterpretAs<Type>](../../sql-reference/functions/type-conversion-functions.md#reinterpretasuint8163264) 函数。
+
+:::note
+如果从 `clickhouse-client` 中调用 `unbin`，则使用 UTF-8 显示二进制字符串。
+:::
+
+支持二进制数字`0`和`1`。二进制位数不必是八的倍数。如果参数字符串包含二进制数字以外的任何内容，则返回一些实现定义的结果（不抛出异常）。
+
+**参数**
+
+- `arg` — 包含任意数量的二进制数字的字符串。类型为[String](../../sql-reference/data-types/string.md)。
+
+**返回值**
+
+- 二进制字符串 (BLOB)。
+
+类型为：[String](../../sql-reference/data-types/string.md)。
+
+**示例**
+
+查询语句：
+
+``` sql
+SELECT UNBIN('001100000011000100110010'), UNBIN('0100110101111001010100110101000101001100');
+```
+
+结果：
+
+``` text
+┌─unbin('001100000011000100110010')─┬─unbin('0100110101111001010100110101000101001100')─┐
+│ 012 │ MySQL │
+└───────────────────────────────────┴───────────────────────────────────────────────────┘
+```
+
+查询语句：
+
+``` sql
+SELECT reinterpretAsUInt64(reverse(unbin('1110'))) AS num;
+```
+
+结果：
+
+``` text
+┌─num─┐
+│ 14 │
+└─────┘
+```
## UUIDStringToNum(str) {#uuidstringtonumstr}
@@ -91,4 +385,55 @@ SELECT char(0xE4, 0xBD, 0xA0, 0xE5, 0xA5, 0xBD) AS hello;
接受一个整数。返回一个UInt64类型数组，其中包含一组2的幂列表，其列表中的所有值相加等于这个整数。数组中的数字按升序排列。
+## bitPositionsToArray(num) {#bitpositionstoarraynum}
+
+接受整数并将其转换为无符号整数。返回一个 `UInt64` 数字数组，其中包含 `arg` 中等于 `1` 的位的位置列表，按升序排列。
+
+**语法**
+
+```sql
+bitPositionsToArray(arg)
+```
+
+**参数**
+
+- `arg` — 整数值。类型为[Int/UInt](../../sql-reference/data-types/int-uint.md)。
+
+**返回值**
+
+- 包含等于 `1` 的位位置列表的数组，按升序排列。
+
+类型为： [Array](../../sql-reference/data-types/array.md)([UInt64](../../sql-reference/data-types/int-uint.md))。
+
+**示例**
+
+查询语句：
+
+``` sql
+SELECT bitPositionsToArray(toInt8(1)) AS bit_positions;
+```
+
+结果：
+
+``` text
+┌─bit_positions─┐
+│ [0] │
+└───────────────┘
+```
+
+查询语句：
+
+``` sql
+SELECT bitPositionsToArray(toInt8(-1)) AS bit_positions;
+```
+
+结果：
+
+``` text
+┌─bit_positions─────┐
+│ [0,1,2,3,4,5,6,7] │
+└───────────────────┘
+```
+
+
[来源文章](https://clickhouse.com/docs/en/query_language/functions/encoding_functions/)
diff --git a/programs/bash-completion/completions/clickhouse-bootstrap b/programs/bash-completion/completions/clickhouse-bootstrap
index 98fcd68db16..8684f122503 100644
--- a/programs/bash-completion/completions/clickhouse-bootstrap
+++ b/programs/bash-completion/completions/clickhouse-bootstrap
@@ -34,6 +34,12 @@ CLICKHOUSE_QueryProcessingStage=(
with_mergeable_state_after_aggregation_and_limit
)
+CLICKHOUSE_QueryKind=(
+ initial_query
+ secondary_query
+ no_query
+)
+
CLICKHOUSE_Format=(
CapnProto
PostgreSQLWire
@@ -124,6 +130,10 @@ function _complete_for_clickhouse_generic_bin_impl()
COMPREPLY=( $(compgen -W "${CLICKHOUSE_QueryProcessingStage[*]}" -- "$cur") )
return 1
;;
+ --query_kind)
+ COMPREPLY=( $(compgen -W "${CLICKHOUSE_QueryKind[*]}" -- "$cur") )
+ return 1
+ ;;
--send_logs_level)
COMPREPLY=( $(compgen -W "${CLICKHOUSE_logs_level[*]}" -- "$cur") )
return 1
diff --git a/programs/client/Client.cpp b/programs/client/Client.cpp
index 4e4e0cc07f5..cbbf195a68c 100644
--- a/programs/client/Client.cpp
+++ b/programs/client/Client.cpp
@@ -1038,6 +1038,7 @@ void Client::processConfig()
ClientInfo & client_info = global_context->getClientInfo();
client_info.setInitialQuery();
client_info.quota_key = config().getString("quota_key", "");
+ client_info.query_kind = query_kind;
}
diff --git a/programs/local/LocalServer.cpp b/programs/local/LocalServer.cpp
index f3fa7ff2bfa..4f3b92bbcf0 100644
--- a/programs/local/LocalServer.cpp
+++ b/programs/local/LocalServer.cpp
@@ -544,8 +544,7 @@ void LocalServer::processConfig()
if (uncompressed_cache_size)
global_context->setUncompressedCache(uncompressed_cache_size);
- /// Size of cache for marks (index of MergeTree family of tables). It is necessary.
- /// Specify default value for mark_cache_size explicitly!
+ /// Size of cache for marks (index of MergeTree family of tables).
size_t mark_cache_size = config().getUInt64("mark_cache_size", 5368709120);
if (mark_cache_size)
global_context->setMarkCache(mark_cache_size);
@@ -555,8 +554,7 @@ void LocalServer::processConfig()
if (index_uncompressed_cache_size)
global_context->setIndexUncompressedCache(index_uncompressed_cache_size);
- /// Size of cache for index marks (index of MergeTree skip indices). It is necessary.
- /// Specify default value for index_mark_cache_size explicitly!
+ /// Size of cache for index marks (index of MergeTree skip indices).
size_t index_mark_cache_size = config().getUInt64("index_mark_cache_size", 0);
if (index_mark_cache_size)
global_context->setIndexMarkCache(index_mark_cache_size);
@@ -626,6 +624,7 @@ void LocalServer::processConfig()
ClientInfo & client_info = global_context->getClientInfo();
client_info.setInitialQuery();
+ client_info.query_kind = query_kind;
}
diff --git a/programs/server/Server.cpp b/programs/server/Server.cpp
index 752ff51ba4f..defc66b0ed9 100644
--- a/programs/server/Server.cpp
+++ b/programs/server/Server.cpp
@@ -1351,8 +1351,8 @@ int Server::main(const std::vector & /*args*/)
settings.async_insert_max_data_size,
AsynchronousInsertQueue::Timeout{.busy = settings.async_insert_busy_timeout_ms, .stale = settings.async_insert_stale_timeout_ms}));
- /// Size of cache for marks (index of MergeTree family of tables). It is mandatory.
- size_t mark_cache_size = config().getUInt64("mark_cache_size");
+ /// Size of cache for marks (index of MergeTree family of tables).
+ size_t mark_cache_size = config().getUInt64("mark_cache_size", 5368709120);
if (!mark_cache_size)
LOG_ERROR(log, "Too low mark cache size will lead to severe performance degradation.");
if (mark_cache_size > max_cache_size)
@@ -1368,8 +1368,7 @@ int Server::main(const std::vector & /*args*/)
if (index_uncompressed_cache_size)
global_context->setIndexUncompressedCache(index_uncompressed_cache_size);
- /// Size of cache for index marks (index of MergeTree skip indices). It is necessary.
- /// Specify default value for index_mark_cache_size explicitly!
+ /// Size of cache for index marks (index of MergeTree skip indices).
size_t index_mark_cache_size = config().getUInt64("index_mark_cache_size", 0);
if (index_mark_cache_size)
global_context->setIndexMarkCache(index_mark_cache_size);
diff --git a/programs/server/config.xml b/programs/server/config.xml
index bd54051be19..203684a9e00 100644
--- a/programs/server/config.xml
+++ b/programs/server/config.xml
@@ -365,6 +365,59 @@
/var/lib/clickhouse/
+
+
+
+
/var/lib/clickhouse/tmp/
@@ -551,6 +604,9 @@
if this setting is true the user B will see all rows, and if this setting is false the user B will see no rows.
By default this setting is false for compatibility with earlier access configurations. -->
false
+
+ false
diff --git a/programs/server/embedded.xml b/programs/server/embedded.xml
index ba0df99dfe0..2b6c4d9f770 100644
--- a/programs/server/embedded.xml
+++ b/programs/server/embedded.xml
@@ -13,7 +13,6 @@
./
8589934592
- 5368709120
true
diff --git a/programs/static-files-disk-uploader/static-files-disk-uploader.cpp b/programs/static-files-disk-uploader/static-files-disk-uploader.cpp
index 20307c0ccd3..a10c25c3342 100644
--- a/programs/static-files-disk-uploader/static-files-disk-uploader.cpp
+++ b/programs/static-files-disk-uploader/static-files-disk-uploader.cpp
@@ -71,7 +71,7 @@ void processFile(const fs::path & file_path, const fs::path & dst_path, bool tes
dst_buf->next();
dst_buf->finalize();
}
-};
+}
void processTableFiles(const fs::path & data_path, fs::path dst_path, bool test_mode, bool link)
diff --git a/src/Access/AccessControl.cpp b/src/Access/AccessControl.cpp
index 141255883b0..81cc0c0f82e 100644
--- a/src/Access/AccessControl.cpp
+++ b/src/Access/AccessControl.cpp
@@ -164,6 +164,10 @@ void AccessControl::setUpFromMainConfig(const Poco::Util::AbstractConfiguration
"access_control_improvements.users_without_row_policies_can_read_rows",
false /* false because we need to be compatible with earlier access configurations */));
+ setOnClusterQueriesRequireClusterGrant(config_.getBool(
+ "access_control_improvements.on_cluster_queries_require_cluster_grant",
+ false /* false because we need to be compatible with earlier access configurations */));
+
addStoragesFromMainConfig(config_, config_path_, get_zookeeper_function_);
checkNoReservedUsersInConfigs();
}
diff --git a/src/Access/AccessControl.h b/src/Access/AccessControl.h
index 1532c7e7719..10180589412 100644
--- a/src/Access/AccessControl.h
+++ b/src/Access/AccessControl.h
@@ -133,6 +133,10 @@ public:
void setEnabledUsersWithoutRowPoliciesCanReadRows(bool enable) { users_without_row_policies_can_read_rows = enable; }
bool isEnabledUsersWithoutRowPoliciesCanReadRows() const { return users_without_row_policies_can_read_rows; }
+ /// Require CLUSTER grant for ON CLUSTER queries.
+ void setOnClusterQueriesRequireClusterGrant(bool enable) { on_cluster_queries_require_cluster_grant = enable; }
+ bool doesOnClusterQueriesRequireClusterGrant() const { return on_cluster_queries_require_cluster_grant; }
+
UUID authenticate(const Credentials & credentials, const Poco::Net::IPAddress & address) const;
void setExternalAuthenticatorsConfig(const Poco::Util::AbstractConfiguration & config);
@@ -190,6 +194,7 @@ private:
std::atomic_bool allow_plaintext_password = true;
std::atomic_bool allow_no_password = true;
std::atomic_bool users_without_row_policies_can_read_rows = false;
+ std::atomic_bool on_cluster_queries_require_cluster_grant = false;
};
}
diff --git a/src/Access/Common/AccessType.h b/src/Access/Common/AccessType.h
index 82dc04db684..8c10fd7e150 100644
--- a/src/Access/Common/AccessType.h
+++ b/src/Access/Common/AccessType.h
@@ -188,6 +188,8 @@ enum class AccessType
M(HIVE, "", GLOBAL, SOURCES) \
M(SOURCES, "", GROUP, ALL) \
\
+ M(CLUSTER, "", GLOBAL, ALL) /* ON CLUSTER queries */ \
+ \
M(ALL, "ALL PRIVILEGES", GROUP, NONE) /* full access */ \
M(NONE, "USAGE, NO PRIVILEGES", GROUP, NONE) /* no access */
diff --git a/src/Access/ContextAccess.cpp b/src/Access/ContextAccess.cpp
index aa4ed6cb41f..28926310c20 100644
--- a/src/Access/ContextAccess.cpp
+++ b/src/Access/ContextAccess.cpp
@@ -359,7 +359,7 @@ std::shared_ptr ContextAccess::getAccessRightsWithImplicit()
template
-bool ContextAccess::checkAccessImplHelper(const AccessFlags & flags, const Args &... args) const
+bool ContextAccess::checkAccessImplHelper(AccessFlags flags, const Args &... args) const
{
auto access_granted = [&]
{
@@ -379,6 +379,9 @@ bool ContextAccess::checkAccessImplHelper(const AccessFlags & flags, const Args
return false;
};
+ if (flags & AccessType::CLUSTER && !access_control->doesOnClusterQueriesRequireClusterGrant())
+ flags &= ~AccessType::CLUSTER;
+
if (!flags || is_full_access)
return access_granted();
diff --git a/src/Access/ContextAccess.h b/src/Access/ContextAccess.h
index 44073320a4c..5742b6a3222 100644
--- a/src/Access/ContextAccess.h
+++ b/src/Access/ContextAccess.h
@@ -179,7 +179,7 @@ private:
bool checkAccessImpl(const AccessRightsElements & elements) const;
template
- bool checkAccessImplHelper(const AccessFlags & flags, const Args &... args) const;
+ bool checkAccessImplHelper(AccessFlags flags, const Args &... args) const;
template
bool checkAccessImplHelper(const AccessRightsElement & element) const;
diff --git a/src/AggregateFunctions/AggregateFunctionMannWhitney.h b/src/AggregateFunctions/AggregateFunctionMannWhitney.h
index 887769dfbf5..089f70cd26b 100644
--- a/src/AggregateFunctions/AggregateFunctionMannWhitney.h
+++ b/src/AggregateFunctions/AggregateFunctionMannWhitney.h
@@ -245,4 +245,4 @@ public:
};
-};
+}
diff --git a/src/AggregateFunctions/AggregateFunctionMeanZTest.h b/src/AggregateFunctions/AggregateFunctionMeanZTest.h
index e4be2503d87..7fecff591e6 100644
--- a/src/AggregateFunctions/AggregateFunctionMeanZTest.h
+++ b/src/AggregateFunctions/AggregateFunctionMeanZTest.h
@@ -136,4 +136,4 @@ public:
}
};
-};
+}
diff --git a/src/AggregateFunctions/AggregateFunctionRankCorrelation.h b/src/AggregateFunctions/AggregateFunctionRankCorrelation.h
index 733416d4721..a9bf8254f35 100644
--- a/src/AggregateFunctions/AggregateFunctionRankCorrelation.h
+++ b/src/AggregateFunctions/AggregateFunctionRankCorrelation.h
@@ -102,4 +102,4 @@ public:
};
-};
+}
diff --git a/src/AggregateFunctions/AggregateFunctionTTest.h b/src/AggregateFunctions/AggregateFunctionTTest.h
index 4c939121a72..7ef5cfce9c9 100644
--- a/src/AggregateFunctions/AggregateFunctionTTest.h
+++ b/src/AggregateFunctions/AggregateFunctionTTest.h
@@ -234,4 +234,4 @@ public:
}
};
-};
+}
diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
index b8c826f1687..8f6e894a100 100644
--- a/src/CMakeLists.txt
+++ b/src/CMakeLists.txt
@@ -37,18 +37,6 @@ if (USE_DEBUG_HELPERS)
add_compile_options($<$:${INCLUDE_DEBUG_HELPERS}>)
endif ()
-if (COMPILER_GCC)
- # If we leave this optimization enabled, gcc-7 replaces a pair of SSE intrinsics (16 byte load, store) with a call to memcpy.
- # It leads to slow code. This is compiler bug. It looks like this:
- #
- # (gdb) bt
- #0 memcpy (destination=0x7faa6e9f1638, source=0x7faa81d9e9a8, size=16) at ../libs/libmemcpy/memcpy.h:11
- #1 0x0000000005341c5f in _mm_storeu_si128 (__B=..., __P=) at /usr/lib/gcc/x86_64-linux-gnu/7/include/emmintrin.h:720
- #2 memcpySmallAllowReadWriteOverflow15Impl (n=, src=, dst=) at ../src/Common/memcpySmall.h:37
-
- add_definitions ("-fno-tree-loop-distribute-patterns")
-endif ()
-
# ClickHouse developers may use platform-dependent code under some macro (e.g. `#ifdef ENABLE_MULTITARGET`).
# If turned ON, this option defines such macro.
# See `src/Common/TargetSpecific.h`
@@ -562,7 +550,7 @@ include ("${ClickHouse_SOURCE_DIR}/cmake/add_check.cmake")
if (ENABLE_TESTS)
macro (grep_gtest_sources BASE_DIR DST_VAR)
# Cold match files that are not in tests/ directories
- file(GLOB_RECURSE "${DST_VAR}" RELATIVE "${BASE_DIR}" "gtest*.cpp")
+ file(GLOB_RECURSE "${DST_VAR}" CONFIGURE_DEPENDS RELATIVE "${BASE_DIR}" "gtest*.cpp")
endmacro()
# attach all dbms gtest sources
diff --git a/src/Client/ClientBase.cpp b/src/Client/ClientBase.cpp
index 35ef55a1387..9cc31df0b43 100644
--- a/src/Client/ClientBase.cpp
+++ b/src/Client/ClientBase.cpp
@@ -119,6 +119,17 @@ namespace ProfileEvents
namespace DB
{
+static ClientInfo::QueryKind parseQueryKind(const String & query_kind)
+{
+ if (query_kind == "initial_query")
+ return ClientInfo::QueryKind::INITIAL_QUERY;
+ if (query_kind == "secondary_query")
+ return ClientInfo::QueryKind::SECONDARY_QUERY;
+ if (query_kind == "no_query")
+ return ClientInfo::QueryKind::NO_QUERY;
+ throw Exception(ErrorCodes::BAD_ARGUMENTS, "Unknown query kind {}", query_kind);
+}
+
static void incrementProfileEventsBlock(Block & dst, const Block & src)
{
if (!dst)
@@ -2125,6 +2136,7 @@ void ClientBase::init(int argc, char ** argv)
("query,q", po::value(), "query")
("stage", po::value()->default_value("complete"), "Request query processing up to specified stage: complete,fetch_columns,with_mergeable_state,with_mergeable_state_after_aggregation,with_mergeable_state_after_aggregation_and_limit")
+ ("query_kind", po::value()->default_value("initial_query"), "One of initial_query/secondary_query/no_query")
("query_id", po::value(), "query_id")
("progress", "print progress of queries execution")
@@ -2255,6 +2267,7 @@ void ClientBase::init(int argc, char ** argv)
server_logs_file = options["server_logs_file"].as();
query_processing_stage = QueryProcessingStage::fromString(options["stage"].as());
+ query_kind = parseQueryKind(options["query_kind"].as());
profile_events.print = options.count("print-profile-events");
profile_events.delay_ms = options["profile-events-delay-ms"].as();
diff --git a/src/Client/ClientBase.h b/src/Client/ClientBase.h
index d373ce5f60b..d11977e984a 100644
--- a/src/Client/ClientBase.h
+++ b/src/Client/ClientBase.h
@@ -256,6 +256,7 @@ protected:
} profile_events;
QueryProcessingStage::Enum query_processing_stage;
+ ClientInfo::QueryKind query_kind;
bool fake_drop = false;
diff --git a/src/Client/Connection.cpp b/src/Client/Connection.cpp
index 96dad3de8ae..5cd286a9710 100644
--- a/src/Client/Connection.cpp
+++ b/src/Client/Connection.cpp
@@ -93,37 +93,58 @@ void Connection::connect(const ConnectionTimeouts & timeouts)
{
try
{
- if (connected)
- disconnect();
-
LOG_TRACE(log_wrapper.get(), "Connecting. Database: {}. User: {}{}{}",
default_database.empty() ? "(not specified)" : default_database,
user,
static_cast(secure) ? ". Secure" : "",
static_cast(compression) ? "" : ". Uncompressed");
- if (static_cast(secure))
- {
-#if USE_SSL
- socket = std::make_unique();
-
- /// we resolve the ip when we open SecureStreamSocket, so to make Server Name Indication (SNI)
- /// work we need to pass host name separately. It will be send into TLS Hello packet to let
- /// the server know which host we want to talk with (single IP can process requests for multiple hosts using SNI).
- static_cast(socket.get())->setPeerHostName(host);
-#else
- throw Exception{"tcp_secure protocol is disabled because poco library was built without NetSSL support.", ErrorCodes::SUPPORT_IS_DISABLED};
-#endif
- }
- else
- {
- socket = std::make_unique();
- }
-
- current_resolved_address = DNSResolver::instance().resolveAddress(host, port);
-
+ auto addresses = DNSResolver::instance().resolveAddressList(host, port);
const auto & connection_timeout = static_cast(secure) ? timeouts.secure_connection_timeout : timeouts.connection_timeout;
- socket->connect(*current_resolved_address, connection_timeout);
+
+ for (auto it = addresses.begin(); it != addresses.end();)
+ {
+ if (connected)
+ disconnect();
+
+ if (static_cast(secure))
+ {
+#if USE_SSL
+ socket = std::make_unique();
+
+ /// we resolve the ip when we open SecureStreamSocket, so to make Server Name Indication (SNI)
+ /// work we need to pass host name separately. It will be send into TLS Hello packet to let
+ /// the server know which host we want to talk with (single IP can process requests for multiple hosts using SNI).
+ static_cast(socket.get())->setPeerHostName(host);
+#else
+ throw Exception{"tcp_secure protocol is disabled because poco library was built without NetSSL support.", ErrorCodes::SUPPORT_IS_DISABLED};
+#endif
+ }
+ else
+ {
+ socket = std::make_unique();
+ }
+
+ try
+ {
+ socket->connect(*it, connection_timeout);
+ current_resolved_address = *it;
+ break;
+ }
+ catch (Poco::Net::NetException &)
+ {
+ if (++it == addresses.end())
+ throw;
+ continue;
+ }
+ catch (Poco::TimeoutException &)
+ {
+ if (++it == addresses.end())
+ throw;
+ continue;
+ }
+ }
+
socket->setReceiveTimeout(timeouts.receive_timeout);
socket->setSendTimeout(timeouts.send_timeout);
socket->setNoDelay(true);
diff --git a/src/Client/HedgedConnections.cpp b/src/Client/HedgedConnections.cpp
index 75f25263b6e..9f0ead79981 100644
--- a/src/Client/HedgedConnections.cpp
+++ b/src/Client/HedgedConnections.cpp
@@ -345,7 +345,7 @@ HedgedConnections::ReplicaLocation HedgedConnections::getReadyReplicaLocation(As
else
throw Exception("Unknown event from epoll", ErrorCodes::LOGICAL_ERROR);
}
-};
+}
bool HedgedConnections::resumePacketReceiver(const HedgedConnections::ReplicaLocation & location)
{
diff --git a/src/Client/LocalConnection.cpp b/src/Client/LocalConnection.cpp
index 77519423763..0707b0bcdc0 100644
--- a/src/Client/LocalConnection.cpp
+++ b/src/Client/LocalConnection.cpp
@@ -73,11 +73,15 @@ void LocalConnection::sendQuery(
const String & query_id,
UInt64 stage,
const Settings *,
- const ClientInfo *,
+ const ClientInfo * client_info,
bool,
std::function process_progress_callback)
{
- query_context = session.makeQueryContext();
+ /// Suggestion comes without client_info.
+ if (client_info)
+ query_context = session.makeQueryContext(*client_info);
+ else
+ query_context = session.makeQueryContext();
query_context->setCurrentQueryId(query_id);
if (send_progress)
{
diff --git a/src/Common/COW.h b/src/Common/COW.h
index f958fe71824..f772acd84e0 100644
--- a/src/Common/COW.h
+++ b/src/Common/COW.h
@@ -219,7 +219,7 @@ protected:
/// Get internal immutable ptr. Does not change internal use counter.
immutable_ptr detach() && { return std::move(value); }
- operator bool() const { return value != nullptr; } /// NOLINT
+ explicit operator bool() const { return value != nullptr; }
bool operator! () const { return value == nullptr; }
bool operator== (const chameleon_ptr & rhs) const { return value == rhs.value; }
diff --git a/src/Common/Config/AbstractConfigurationComparison.cpp b/src/Common/Config/AbstractConfigurationComparison.cpp
index ea0b3be4b98..711c754743d 100644
--- a/src/Common/Config/AbstractConfigurationComparison.cpp
+++ b/src/Common/Config/AbstractConfigurationComparison.cpp
@@ -18,7 +18,7 @@ namespace
result += '.';
result += subkey;
return result;
- };
+ }
}
diff --git a/src/Common/Config/configReadClient.cpp b/src/Common/Config/configReadClient.cpp
index e7bc0b72814..e5308bc3bc7 100644
--- a/src/Common/Config/configReadClient.cpp
+++ b/src/Common/Config/configReadClient.cpp
@@ -14,7 +14,7 @@ bool safeFsExists(const String & path)
{
std::error_code ec;
return fs::exists(path, ec);
-};
+}
bool configReadClient(Poco::Util::LayeredConfiguration & config, const std::string & home_path)
{
diff --git a/src/Common/DNSResolver.cpp b/src/Common/DNSResolver.cpp
index d757ec2ae2a..0616e324b73 100644
--- a/src/Common/DNSResolver.cpp
+++ b/src/Common/DNSResolver.cpp
@@ -83,25 +83,8 @@ static void splitHostAndPort(const std::string & host_and_port, std::string & ou
throw Exception("Port must be numeric", ErrorCodes::BAD_ARGUMENTS);
}
-static DNSResolver::IPAddresses resolveIPAddressImpl(const std::string & host)
+static DNSResolver::IPAddresses hostByName(const std::string & host)
{
- Poco::Net::IPAddress ip;
-
- /// NOTE:
- /// - Poco::Net::DNS::resolveOne(host) doesn't work for IP addresses like 127.0.0.2
- /// - Poco::Net::IPAddress::tryParse() expect hex string for IPv6 (without brackets)
- if (host.starts_with('['))
- {
- assert(host.ends_with(']'));
- if (Poco::Net::IPAddress::tryParse(host.substr(1, host.size() - 2), ip))
- return DNSResolver::IPAddresses(1, ip);
- }
- else
- {
- if (Poco::Net::IPAddress::tryParse(host, ip))
- return DNSResolver::IPAddresses(1, ip);
- }
-
/// Family: AF_UNSPEC
/// AI_ALL is required for checking if client is allowed to connect from an address
auto flags = Poco::Net::DNS::DNS_HINT_AI_V4MAPPED | Poco::Net::DNS::DNS_HINT_AI_ALL;
@@ -131,6 +114,30 @@ static DNSResolver::IPAddresses resolveIPAddressImpl(const std::string & host)
return addresses;
}
+static DNSResolver::IPAddresses resolveIPAddressImpl(const std::string & host)
+{
+ Poco::Net::IPAddress ip;
+
+ /// NOTE:
+ /// - Poco::Net::DNS::resolveOne(host) doesn't work for IP addresses like 127.0.0.2
+ /// - Poco::Net::IPAddress::tryParse() expect hex string for IPv6 (without brackets)
+ if (host.starts_with('['))
+ {
+ assert(host.ends_with(']'));
+ if (Poco::Net::IPAddress::tryParse(host.substr(1, host.size() - 2), ip))
+ return DNSResolver::IPAddresses(1, ip);
+ }
+ else
+ {
+ if (Poco::Net::IPAddress::tryParse(host, ip))
+ return DNSResolver::IPAddresses(1, ip);
+ }
+
+ DNSResolver::IPAddresses addresses = hostByName(host);
+
+ return addresses;
+}
+
static String reverseResolveImpl(const Poco::Net::IPAddress & address)
{
Poco::Net::SocketAddress sock_addr(address, 0);
@@ -208,6 +215,26 @@ Poco::Net::SocketAddress DNSResolver::resolveAddress(const std::string & host, U
return Poco::Net::SocketAddress(impl->cache_host(host).front(), port);
}
+std::vector DNSResolver::resolveAddressList(const std::string & host, UInt16 port)
+{
+ if (Poco::Net::IPAddress ip; Poco::Net::IPAddress::tryParse(host, ip))
+ return std::vector{{ip, port}};
+
+ std::vector addresses;
+
+ if (!impl->disable_cache)
+ addToNewHosts(host);
+
+ std::vector ips = impl->disable_cache ? hostByName(host) : impl->cache_host(host);
+ auto ips_end = std::unique(ips.begin(), ips.end());
+
+ addresses.reserve(ips_end - ips.begin());
+ for (auto ip = ips.begin(); ip != ips_end; ++ip)
+ addresses.emplace_back(*ip, port);
+
+ return addresses;
+}
+
String DNSResolver::reverseResolve(const Poco::Net::IPAddress & address)
{
if (impl->disable_cache)
diff --git a/src/Common/DNSResolver.h b/src/Common/DNSResolver.h
index 3f1773d4050..fdd9799f96f 100644
--- a/src/Common/DNSResolver.h
+++ b/src/Common/DNSResolver.h
@@ -34,6 +34,8 @@ public:
Poco::Net::SocketAddress resolveAddress(const std::string & host, UInt16 port);
+ std::vector resolveAddressList(const std::string & host, UInt16 port);
+
/// Accepts host IP and resolves its host name
String reverseResolve(const Poco::Net::IPAddress & address);
diff --git a/src/Common/Exception.h b/src/Common/Exception.h
index b2fc369237e..086b64bf5f9 100644
--- a/src/Common/Exception.h
+++ b/src/Common/Exception.h
@@ -48,8 +48,8 @@ public:
Exception * clone() const override { return new Exception(*this); }
void rethrow() const override { throw *this; }
- const char * name() const throw() override { return "DB::Exception"; }
- const char * what() const throw() override { return message().data(); }
+ const char * name() const noexcept override { return "DB::Exception"; }
+ const char * what() const noexcept override { return message().data(); }
/// Add something to the existing message.
template
@@ -77,7 +77,7 @@ private:
#endif
bool remote = false;
- const char * className() const throw() override { return "DB::Exception"; }
+ const char * className() const noexcept override { return "DB::Exception"; }
};
@@ -102,8 +102,8 @@ private:
int saved_errno;
std::optional path;
- const char * name() const throw() override { return "DB::ErrnoException"; }
- const char * className() const throw() override { return "DB::ErrnoException"; }
+ const char * name() const noexcept override { return "DB::ErrnoException"; }
+ const char * className() const noexcept override { return "DB::ErrnoException"; }
};
@@ -143,8 +143,8 @@ private:
String file_name;
mutable std::string formatted_message;
- const char * name() const throw() override { return "DB::ParsingException"; }
- const char * className() const throw() override { return "DB::ParsingException"; }
+ const char * name() const noexcept override { return "DB::ParsingException"; }
+ const char * className() const noexcept override { return "DB::ParsingException"; }
};
diff --git a/src/Common/HashTable/StringHashTable.h b/src/Common/HashTable/StringHashTable.h
index 7e259d66cd0..6a8bdc06218 100644
--- a/src/Common/HashTable/StringHashTable.h
+++ b/src/Common/HashTable/StringHashTable.h
@@ -169,7 +169,7 @@ struct StringHashTableLookupResult
auto & operator*() const { return *this; }
auto * operator->() { return this; }
auto * operator->() const { return this; }
- operator bool() const { return mapped_ptr; } /// NOLINT
+ explicit operator bool() const { return mapped_ptr; }
friend bool operator==(const StringHashTableLookupResult & a, const std::nullptr_t &) { return !a.mapped_ptr; }
friend bool operator==(const std::nullptr_t &, const StringHashTableLookupResult & b) { return !b.mapped_ptr; }
friend bool operator!=(const StringHashTableLookupResult & a, const std::nullptr_t &) { return a.mapped_ptr; }
diff --git a/src/Common/NetException.h b/src/Common/NetException.h
index 019a12f23b9..712893ed83b 100644
--- a/src/Common/NetException.h
+++ b/src/Common/NetException.h
@@ -22,8 +22,8 @@ public:
void rethrow() const override { throw *this; }
private:
- const char * name() const throw() override { return "DB::NetException"; }
- const char * className() const throw() override { return "DB::NetException"; }
+ const char * name() const noexcept override { return "DB::NetException"; }
+ const char * className() const noexcept override { return "DB::NetException"; }
};
}
diff --git a/src/Common/SensitiveDataMasker.h b/src/Common/SensitiveDataMasker.h
index edd9f10ca91..adb6f5d51e1 100644
--- a/src/Common/SensitiveDataMasker.h
+++ b/src/Common/SensitiveDataMasker.h
@@ -69,4 +69,4 @@ public:
size_t rulesCount() const;
};
-};
+}
diff --git a/src/Common/TargetSpecific.h b/src/Common/TargetSpecific.h
index 2b81ee2fcb3..89c0f467fe3 100644
--- a/src/Common/TargetSpecific.h
+++ b/src/Common/TargetSpecific.h
@@ -276,7 +276,7 @@ DECLARE_AVX512F_SPECIFIC_CODE(
\
FUNCTION_HEADER \
\
- AVX2_FUNCTION_SPECIFIC_ATTRIBUTE \
+ SSE42_FUNCTION_SPECIFIC_ATTRIBUTE \
name##SSE42 \
FUNCTION_BODY \
\
diff --git a/src/Common/ZooKeeper/IKeeper.h b/src/Common/ZooKeeper/IKeeper.h
index 74b45d411b0..1e79468b7e3 100644
--- a/src/Common/ZooKeeper/IKeeper.h
+++ b/src/Common/ZooKeeper/IKeeper.h
@@ -401,8 +401,8 @@ public:
Exception(const Error code_, const std::string & path); /// NOLINT
Exception(const Exception & exc);
- const char * name() const throw() override { return "Coordination::Exception"; }
- const char * className() const throw() override { return "Coordination::Exception"; }
+ const char * name() const noexcept override { return "Coordination::Exception"; }
+ const char * className() const noexcept override { return "Coordination::Exception"; }
Exception * clone() const override { return new Exception(*this); }
const Error code;
diff --git a/src/Common/tests/gtest_lru_file_cache.cpp b/src/Common/tests/gtest_lru_file_cache.cpp
index 24e69259241..36137e02a84 100644
--- a/src/Common/tests/gtest_lru_file_cache.cpp
+++ b/src/Common/tests/gtest_lru_file_cache.cpp
@@ -32,7 +32,7 @@ void assertRange(
ASSERT_EQ(range.left, expected_range.left);
ASSERT_EQ(range.right, expected_range.right);
ASSERT_EQ(file_segment->state(), expected_state);
-};
+}
void printRanges(const auto & segments)
{
diff --git a/src/Common/tests/gtest_sensitive_data_masker.cpp b/src/Common/tests/gtest_sensitive_data_masker.cpp
index 7ebf141d961..b9ee9025c03 100644
--- a/src/Common/tests/gtest_sensitive_data_masker.cpp
+++ b/src/Common/tests/gtest_sensitive_data_masker.cpp
@@ -22,7 +22,7 @@ extern const int CANNOT_COMPILE_REGEXP;
extern const int NO_ELEMENTS_IN_CONFIG;
extern const int INVALID_CONFIG_PARAMETER;
}
-};
+}
TEST(Common, SensitiveDataMasker)
diff --git a/src/Compression/CompressedWriteBuffer.cpp b/src/Compression/CompressedWriteBuffer.cpp
index 2ade1ef0949..93f163dc1af 100644
--- a/src/Compression/CompressedWriteBuffer.cpp
+++ b/src/Compression/CompressedWriteBuffer.cpp
@@ -1,11 +1,11 @@
#include
#include
-#include
#include
+#include
-#include "CompressedWriteBuffer.h"
#include
+#include "CompressedWriteBuffer.h"
namespace DB
@@ -22,14 +22,29 @@ void CompressedWriteBuffer::nextImpl()
if (!offset())
return;
+ UInt32 compressed_size = 0;
size_t decompressed_size = offset();
UInt32 compressed_reserve_size = codec->getCompressedReserveSize(decompressed_size);
- compressed_buffer.resize(compressed_reserve_size);
- UInt32 compressed_size = codec->compress(working_buffer.begin(), decompressed_size, compressed_buffer.data());
- CityHash_v1_0_2::uint128 checksum = CityHash_v1_0_2::CityHash128(compressed_buffer.data(), compressed_size);
- out.write(reinterpret_cast(&checksum), CHECKSUM_SIZE);
- out.write(compressed_buffer.data(), compressed_size);
+ if (out.available() > compressed_reserve_size + CHECKSUM_SIZE)
+ {
+ char * out_checksum_ptr = out.position();
+ char * out_compressed_ptr = out.position() + CHECKSUM_SIZE;
+ compressed_size = codec->compress(working_buffer.begin(), decompressed_size, out_compressed_ptr);
+
+ CityHash_v1_0_2::uint128 checksum = CityHash_v1_0_2::CityHash128(out_compressed_ptr, compressed_size);
+ memcpy(out_checksum_ptr, reinterpret_cast(&checksum), CHECKSUM_SIZE);
+ out.position() += CHECKSUM_SIZE + compressed_size;
+ }
+ else
+ {
+ compressed_buffer.resize(compressed_reserve_size);
+ compressed_size = codec->compress(working_buffer.begin(), decompressed_size, compressed_buffer.data());
+
+ CityHash_v1_0_2::uint128 checksum = CityHash_v1_0_2::CityHash128(compressed_buffer.data(), compressed_size);
+ out.write(reinterpret_cast(&checksum), CHECKSUM_SIZE);
+ out.write(compressed_buffer.data(), compressed_size);
+ }
}
CompressedWriteBuffer::~CompressedWriteBuffer()
@@ -37,10 +52,7 @@ CompressedWriteBuffer::~CompressedWriteBuffer()
finalize();
}
-CompressedWriteBuffer::CompressedWriteBuffer(
- WriteBuffer & out_,
- CompressionCodecPtr codec_,
- size_t buf_size)
+CompressedWriteBuffer::CompressedWriteBuffer(WriteBuffer & out_, CompressionCodecPtr codec_, size_t buf_size)
: BufferWithOwnMemory(buf_size), out(out_), codec(std::move(codec_))
{
}
diff --git a/src/Compression/tests/gtest_compressionCodec.cpp b/src/Compression/tests/gtest_compressionCodec.cpp
index 2d26cfcd5e1..77050908265 100644
--- a/src/Compression/tests/gtest_compressionCodec.cpp
+++ b/src/Compression/tests/gtest_compressionCodec.cpp
@@ -790,7 +790,7 @@ std::vector generatePyramidOfSequences(const size_t sequences
}
return sequences;
-};
+}
// helper macro to produce human-friendly sequence name from generator
#define G(generator) generator, #generator
diff --git a/src/Coordination/KeeperServer.cpp b/src/Coordination/KeeperServer.cpp
index 6961f31ed20..1f089ba2cb7 100644
--- a/src/Coordination/KeeperServer.cpp
+++ b/src/Coordination/KeeperServer.cpp
@@ -15,6 +15,7 @@
#include
#include
#include
+#include
#include
#include
#include
@@ -315,6 +316,22 @@ void KeeperServer::startup(const Poco::Util::AbstractConfiguration & config, boo
state_manager->loadLogStore(state_machine->last_commit_index() + 1, coordination_settings->reserved_log_items);
+ auto log_store = state_manager->load_log_store();
+ auto next_log_idx = log_store->next_slot();
+ if (next_log_idx > 0 && next_log_idx > state_machine->last_commit_index())
+ {
+ auto log_entries = log_store->log_entries(state_machine->last_commit_index() + 1, next_log_idx);
+
+ auto idx = state_machine->last_commit_index() + 1;
+ for (const auto & entry : *log_entries)
+ {
+ if (entry && entry->get_val_type() == nuraft::log_val_type::app_log)
+ state_machine->preprocess(idx, entry->get_buf());
+
+ ++idx;
+ }
+ }
+
loadLatestConfig();
last_local_config = state_manager->parseServersConfiguration(config, true).cluster_config;
diff --git a/src/Coordination/KeeperStateMachine.cpp b/src/Coordination/KeeperStateMachine.cpp
index be7110fa841..fa3a5195226 100644
--- a/src/Coordination/KeeperStateMachine.cpp
+++ b/src/Coordination/KeeperStateMachine.cpp
@@ -44,7 +44,6 @@ namespace
else /// backward compatibility
request_for_session.time = std::chrono::duration_cast(std::chrono::system_clock::now().time_since_epoch()).count();
-
return request_for_session;
}
}
@@ -114,6 +113,21 @@ void KeeperStateMachine::init()
storage = std::make_unique(coordination_settings->dead_session_check_period_ms.totalMilliseconds(), superdigest);
}
+nuraft::ptr KeeperStateMachine::pre_commit(uint64_t log_idx, nuraft::buffer & data)
+{
+ preprocess(log_idx, data);
+ return nullptr;
+}
+
+void KeeperStateMachine::preprocess(const uint64_t log_idx, nuraft::buffer & data)
+{
+ auto request_for_session = parseRequest(data);
+ if (request_for_session.request->getOpNum() == Coordination::OpNum::SessionID)
+ return;
+ std::lock_guard lock(storage_and_responses_lock);
+ storage->preprocessRequest(request_for_session.request, request_for_session.session_id, request_for_session.time, log_idx);
+}
+
nuraft::ptr KeeperStateMachine::commit(const uint64_t log_idx, nuraft::buffer & data)
{
auto request_for_session = parseRequest(data);
@@ -182,6 +196,12 @@ void KeeperStateMachine::commit_config(const uint64_t /* log_idx */, nuraft::ptr
cluster_config = ClusterConfig::deserialize(*tmp);
}
+void KeeperStateMachine::rollback(uint64_t log_idx, nuraft::buffer & /*data*/)
+{
+ std::lock_guard lock(storage_and_responses_lock);
+ storage->rollbackRequest(log_idx);
+}
+
nuraft::ptr KeeperStateMachine::last_snapshot()
{
/// Just return the latest snapshot.
@@ -343,7 +363,7 @@ void KeeperStateMachine::processReadRequest(const KeeperStorage::RequestForSessi
{
/// Pure local request, just process it with storage
std::lock_guard lock(storage_and_responses_lock);
- auto responses = storage->processRequest(request_for_session.request, request_for_session.session_id, request_for_session.time, std::nullopt);
+ auto responses = storage->processRequest(request_for_session.request, request_for_session.session_id, request_for_session.time, std::nullopt, true /*check_acl*/, true /*is_local*/);
for (const auto & response : responses)
if (!responses_queue.push(response))
throw Exception(ErrorCodes::SYSTEM_ERROR, "Could not push response with session id {} into responses queue", response.session_id);
diff --git a/src/Coordination/KeeperStateMachine.h b/src/Coordination/KeeperStateMachine.h
index 73578e6a2ba..aed96a59c13 100644
--- a/src/Coordination/KeeperStateMachine.h
+++ b/src/Coordination/KeeperStateMachine.h
@@ -27,16 +27,16 @@ public:
/// Read state from the latest snapshot
void init();
- /// Currently not supported
- nuraft::ptr pre_commit(const uint64_t /*log_idx*/, nuraft::buffer & /*data*/) override { return nullptr; }
+ void preprocess(uint64_t log_idx, nuraft::buffer & data);
+
+ nuraft::ptr pre_commit(uint64_t log_idx, nuraft::buffer & data) override;
nuraft::ptr commit(const uint64_t log_idx, nuraft::buffer & data) override; /// NOLINT
/// Save new cluster config to our snapshot (copy of the config stored in StateManager)
void commit_config(const uint64_t log_idx, nuraft::ptr & new_conf) override; /// NOLINT
- /// Currently not supported
- void rollback(const uint64_t /*log_idx*/, nuraft::buffer & /*data*/) override {}
+ void rollback(uint64_t log_idx, nuraft::buffer & data) override;
uint64_t last_commit_index() override { return last_committed_idx; }
diff --git a/src/Coordination/KeeperStorage.cpp b/src/Coordination/KeeperStorage.cpp
index f58776cf843..6c0699be95c 100644
--- a/src/Coordination/KeeperStorage.cpp
+++ b/src/Coordination/KeeperStorage.cpp
@@ -1,19 +1,21 @@
-#include
-#include
-#include
-#include
-#include
-#include
-#include
-#include
-#include
-#include
-#include
-#include
-#include
-#include
#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include "Common/ZooKeeper/ZooKeeperConstants.h"
+#include
+#include
+#include
#include
+#include
namespace DB
{
@@ -49,37 +51,10 @@ String getSHA1(const String & userdata)
String generateDigest(const String & userdata)
{
std::vector user_password;
- boost::split(user_password, userdata, [](char c) { return c == ':'; });
+ boost::split(user_password, userdata, [](char character) { return character == ':'; });
return user_password[0] + ":" + base64Encode(getSHA1(userdata));
}
-bool checkACL(int32_t permission, const Coordination::ACLs & node_acls, const std::vector & session_auths)
-{
- if (node_acls.empty())
- return true;
-
- for (const auto & session_auth : session_auths)
- if (session_auth.scheme == "super")
- return true;
-
- for (const auto & node_acl : node_acls)
- {
- if (node_acl.permissions & permission)
- {
- if (node_acl.scheme == "world" && node_acl.id == "anyone")
- return true;
-
- for (const auto & session_auth : session_auths)
- {
- if (node_acl.scheme == session_auth.scheme && node_acl.id == session_auth.id)
- return true;
- }
- }
- }
-
- return false;
-}
-
bool fixupACL(
const std::vector & request_acls,
const std::vector & current_ids,
@@ -122,11 +97,12 @@ bool fixupACL(
return valid_found;
}
-KeeperStorage::ResponsesForSessions processWatchesImpl(const String & path, KeeperStorage::Watches & watches, KeeperStorage::Watches & list_watches, Coordination::Event event_type)
+KeeperStorage::ResponsesForSessions processWatchesImpl(
+ const String & path, KeeperStorage::Watches & watches, KeeperStorage::Watches & list_watches, Coordination::Event event_type)
{
KeeperStorage::ResponsesForSessions result;
- auto it = watches.find(path);
- if (it != watches.end())
+ auto watch_it = watches.find(path);
+ if (watch_it != watches.end())
{
std::shared_ptr watch_response = std::make_shared();
watch_response->path = path;
@@ -134,10 +110,10 @@ KeeperStorage::ResponsesForSessions processWatchesImpl(const String & path, Keep
watch_response->zxid = -1;
watch_response->type = event_type;
watch_response->state = Coordination::State::CONNECTED;
- for (auto watcher_session : it->second)
+ for (auto watcher_session : watch_it->second)
result.push_back(KeeperStorage::ResponseForSession{watcher_session, watch_response});
- watches.erase(it);
+ watches.erase(watch_it);
}
auto parent_path = parentPath(path);
@@ -156,10 +132,11 @@ KeeperStorage::ResponsesForSessions processWatchesImpl(const String & path, Keep
for (const auto & path_to_check : paths_to_check_for_list_watches)
{
- it = list_watches.find(path_to_check);
- if (it != list_watches.end())
+ watch_it = list_watches.find(path_to_check);
+ if (watch_it != list_watches.end())
{
- std::shared_ptr watch_list_response = std::make_shared();
+ std::shared_ptr watch_list_response
+ = std::make_shared();
watch_list_response->path = path_to_check;
watch_list_response->xid = Coordination::WATCH_XID;
watch_list_response->zxid = -1;
@@ -169,14 +146,15 @@ KeeperStorage::ResponsesForSessions processWatchesImpl(const String & path, Keep
watch_list_response->type = Coordination::Event::DELETED;
watch_list_response->state = Coordination::State::CONNECTED;
- for (auto watcher_session : it->second)
+ for (auto watcher_session : watch_it->second)
result.push_back(KeeperStorage::ResponseForSession{watcher_session, watch_list_response});
- list_watches.erase(it);
+ list_watches.erase(watch_it);
}
}
return result;
}
+
}
void KeeperStorage::Node::setData(String new_data)
@@ -198,24 +176,322 @@ void KeeperStorage::Node::removeChild(StringRef child_path)
}
KeeperStorage::KeeperStorage(int64_t tick_time_ms, const String & superdigest_)
- : session_expiry_queue(tick_time_ms)
- , superdigest(superdigest_)
+ : session_expiry_queue(tick_time_ms), superdigest(superdigest_)
{
container.insert("/", Node());
}
-using Undo = std::function;
+template
+struct Overloaded : Ts...
+{
+ using Ts::operator()...;
+};
+
+// explicit deduction guide
+// https://en.cppreference.com/w/cpp/language/class_template_argument_deduction
+template
+Overloaded(Ts...) -> Overloaded;
+
+std::shared_ptr KeeperStorage::UncommittedState::getNode(StringRef path)
+{
+ std::shared_ptr node{nullptr};
+
+ if (auto maybe_node_it = storage.container.find(path); maybe_node_it != storage.container.end())
+ {
+ const auto & committed_node = maybe_node_it->value;
+ node = std::make_shared();
+ node->stat = committed_node.stat;
+ node->seq_num = committed_node.seq_num;
+ node->setData(committed_node.getData());
+ }
+
+ applyDeltas(
+ path,
+ Overloaded{
+ [&](const CreateNodeDelta & create_delta)
+ {
+ assert(!node);
+ node = std::make_shared();
+ node->stat = create_delta.stat;
+ node->setData(create_delta.data);
+ },
+ [&](const RemoveNodeDelta & /*remove_delta*/)
+ {
+ assert(node);
+ node = nullptr;
+ },
+ [&](const UpdateNodeDelta & update_delta)
+ {
+ assert(node);
+ update_delta.update_fn(*node);
+ },
+ [&](auto && /*delta*/) {},
+ });
+
+ return node;
+}
+
+bool KeeperStorage::UncommittedState::hasNode(StringRef path) const
+{
+ bool exists = storage.container.contains(std::string{path});
+ applyDeltas(
+ path,
+ Overloaded{
+ [&](const CreateNodeDelta & /*create_delta*/)
+ {
+ assert(!exists);
+ exists = true;
+ },
+ [&](const RemoveNodeDelta & /*remove_delta*/)
+ {
+ assert(exists);
+ exists = false;
+ },
+ [&](auto && /*delta*/) {},
+ });
+
+ return exists;
+}
+
+Coordination::ACLs KeeperStorage::UncommittedState::getACLs(StringRef path) const
+{
+ std::optional acl_id;
+ if (auto maybe_node_it = storage.container.find(path); maybe_node_it != storage.container.end())
+ acl_id.emplace(maybe_node_it->value.acl_id);
+
+ const Coordination::ACLs * acls{nullptr};
+ applyDeltas(
+ path,
+ Overloaded{
+ [&](const CreateNodeDelta & create_delta)
+ {
+ assert(!acl_id);
+ acls = &create_delta.acls;
+ },
+ [&](const RemoveNodeDelta & /*remove_delta*/)
+ {
+ assert(acl_id || acls);
+ acl_id.reset();
+ acls = nullptr;
+ },
+ [&](const SetACLDelta & set_acl_delta)
+ {
+ assert(acl_id || acls);
+ acls = &set_acl_delta.acls;
+ },
+ [&](auto && /*delta*/) {},
+ });
+
+ if (acls)
+ return *acls;
+
+ return acl_id ? storage.acl_map.convertNumber(*acl_id) : Coordination::ACLs{};
+}
+
+namespace
+{
+
+[[noreturn]] void onStorageInconsistency()
+{
+ LOG_ERROR(&Poco::Logger::get("KeeperStorage"), "Inconsistency found between uncommitted and committed data. Keeper will terminate to avoid undefined behaviour.");
+ std::terminate();
+}
+
+}
+
+Coordination::Error KeeperStorage::commit(int64_t commit_zxid, int64_t session_id)
+{
+ // Deltas are added with increasing ZXIDs
+ // If there are no deltas for the commit_zxid (e.g. read requests), we instantly return
+ // on first delta
+ for (auto & delta : uncommitted_state.deltas)
+ {
+ if (delta.zxid > commit_zxid)
+ break;
+
+ bool finish_subdelta = false;
+ auto result = std::visit(
+ [&, &path = delta.path](DeltaType & operation) -> Coordination::Error
+ {
+ if constexpr (std::same_as)
+ {
+ if (!createNode(
+ path,
+ std::move(operation.data),
+ operation.stat,
+ operation.is_sequental,
+ operation.is_ephemeral,
+ std::move(operation.acls),
+ session_id))
+ onStorageInconsistency();
+
+ return Coordination::Error::ZOK;
+ }
+ else if constexpr (std::same_as)
+ {
+ auto node_it = container.find(path);
+ if (node_it == container.end())
+ onStorageInconsistency();
+
+ if (operation.version != -1 && operation.version != node_it->value.stat.version)
+ onStorageInconsistency();
+
+ container.updateValue(path, operation.update_fn);
+ return Coordination::Error::ZOK;
+ }
+ else if constexpr (std::same_as)
+ {
+ if (!removeNode(path, operation.version))
+ onStorageInconsistency();
+
+ return Coordination::Error::ZOK;
+ }
+ else if constexpr (std::same_as)
+ {
+ auto node_it = container.find(path);
+ if (node_it == container.end())
+ onStorageInconsistency();
+
+ if (operation.version != -1 && operation.version != node_it->value.stat.aversion)
+ onStorageInconsistency();
+
+ acl_map.removeUsage(node_it->value.acl_id);
+
+ uint64_t acl_id = acl_map.convertACLs(operation.acls);
+ acl_map.addUsage(acl_id);
+
+ container.updateValue(path, [acl_id](KeeperStorage::Node & node) { node.acl_id = acl_id; });
+
+ return Coordination::Error::ZOK;
+ }
+ else if constexpr (std::same_as)
+ return operation.error;
+ else if constexpr (std::same_as)
+ {
+ finish_subdelta = true;
+ return Coordination::Error::ZOK;
+ }
+ else if constexpr (std::same_as)
+ {
+ session_and_auth[operation.session_id].emplace_back(std::move(operation.auth_id));
+ return Coordination::Error::ZOK;
+ }
+ else
+ {
+ // shouldn't be called in any process functions
+ onStorageInconsistency();
+ }
+ },
+ delta.operation);
+
+ if (result != Coordination::Error::ZOK)
+ return result;
+
+ if (finish_subdelta)
+ return Coordination::Error::ZOK;
+ }
+
+ return Coordination::Error::ZOK;
+}
+
+bool KeeperStorage::createNode(
+ const std::string & path,
+ String data,
+ const Coordination::Stat & stat,
+ bool is_sequental,
+ bool is_ephemeral,
+ Coordination::ACLs node_acls,
+ int64_t session_id)
+{
+ auto parent_path = parentPath(path);
+ auto node_it = container.find(parent_path);
+
+ if (node_it == container.end())
+ return false;
+
+ if (node_it->value.stat.ephemeralOwner != 0)
+ return false;
+
+ if (container.contains(path))
+ return false;
+
+ KeeperStorage::Node created_node;
+
+ uint64_t acl_id = acl_map.convertACLs(node_acls);
+ acl_map.addUsage(acl_id);
+
+ created_node.acl_id = acl_id;
+ created_node.stat = stat;
+ created_node.setData(std::move(data));
+ created_node.is_sequental = is_sequental;
+ auto [map_key, _] = container.insert(path, created_node);
+ /// Take child path from key owned by map.
+ auto child_path = getBaseName(map_key->getKey());
+ container.updateValue(parent_path, [child_path](KeeperStorage::Node & parent) { parent.addChild(child_path); });
+
+ if (is_ephemeral)
+ ephemerals[session_id].emplace(path);
+
+ return true;
+};
+
+bool KeeperStorage::removeNode(const std::string & path, int32_t version)
+{
+ auto node_it = container.find(path);
+ if (node_it == container.end())
+ return false;
+
+ if (version != -1 && version != node_it->value.stat.version)
+ return false;
+
+ if (node_it->value.stat.numChildren)
+ return false;
+
+ auto prev_node = node_it->value;
+ if (prev_node.stat.ephemeralOwner != 0)
+ {
+ auto ephemerals_it = ephemerals.find(prev_node.stat.ephemeralOwner);
+ ephemerals_it->second.erase(path);
+ if (ephemerals_it->second.empty())
+ ephemerals.erase(ephemerals_it);
+ }
+
+ acl_map.removeUsage(prev_node.acl_id);
+
+ container.updateValue(
+ parentPath(path),
+ [child_basename = getBaseName(node_it->key)](KeeperStorage::Node & parent) { parent.removeChild(child_basename); });
+
+ container.erase(path);
+ return true;
+}
+
struct KeeperStorageRequestProcessor
{
Coordination::ZooKeeperRequestPtr zk_request;
- explicit KeeperStorageRequestProcessor(const Coordination::ZooKeeperRequestPtr & zk_request_)
- : zk_request(zk_request_)
- {}
- virtual std::pair process(KeeperStorage & storage, int64_t zxid, int64_t session_id, int64_t time) const = 0;
- virtual KeeperStorage::ResponsesForSessions processWatches(KeeperStorage::Watches & /*watches*/, KeeperStorage::Watches & /*list_watches*/) const { return {}; }
- virtual bool checkAuth(KeeperStorage & /*storage*/, int64_t /*session_id*/) const { return true; }
+ explicit KeeperStorageRequestProcessor(const Coordination::ZooKeeperRequestPtr & zk_request_) : zk_request(zk_request_) { }
+ virtual Coordination::ZooKeeperResponsePtr process(KeeperStorage & storage, int64_t zxid, int64_t session_id, int64_t time) const = 0;
+ virtual std::vector
+ preprocess(KeeperStorage & /*storage*/, int64_t /*zxid*/, int64_t /*session_id*/, int64_t /*time*/) const
+ {
+ return {};
+ }
+
+ // process the request using locally committed data
+ virtual Coordination::ZooKeeperResponsePtr
+ processLocal(KeeperStorage & /*storage*/, int64_t /*zxid*/, int64_t /*session_id*/, int64_t /*time*/) const
+ {
+ throw Exception{DB::ErrorCodes::LOGICAL_ERROR, "Cannot process the request locally"};
+ }
+
+ virtual KeeperStorage::ResponsesForSessions
+ processWatches(KeeperStorage::Watches & /*watches*/, KeeperStorage::Watches & /*list_watches*/) const
+ {
+ return {};
+ }
+ virtual bool checkAuth(KeeperStorage & /*storage*/, int64_t /*session_id*/, bool /*is_local*/) const { return true; }
virtual ~KeeperStorageRequestProcessor() = default;
};
@@ -223,331 +499,328 @@ struct KeeperStorageRequestProcessor
struct KeeperStorageHeartbeatRequestProcessor final : public KeeperStorageRequestProcessor
{
using KeeperStorageRequestProcessor::KeeperStorageRequestProcessor;
- std::pair process(KeeperStorage & /* storage */, int64_t /* zxid */, int64_t /* session_id */, int64_t /* time */) const override
+ Coordination::ZooKeeperResponsePtr
+ process(KeeperStorage & /* storage */, int64_t /* zxid */, int64_t /* session_id */, int64_t /* time */) const override
{
- return {zk_request->makeResponse(), {}};
+ return zk_request->makeResponse();
}
};
struct KeeperStorageSyncRequestProcessor final : public KeeperStorageRequestProcessor
{
using KeeperStorageRequestProcessor::KeeperStorageRequestProcessor;
- std::pair process(KeeperStorage & /* storage */, int64_t /* zxid */, int64_t /* session_id */, int64_t /* time */) const override
+ Coordination::ZooKeeperResponsePtr
+ process(KeeperStorage & /* storage */, int64_t /* zxid */, int64_t /* session_id */, int64_t /* time */) const override
{
auto response = zk_request->makeResponse();
dynamic_cast(*response).path
= dynamic_cast(*zk_request).path;
- return {response, {}};
- }
-};
-
-struct KeeperStorageCreateRequestProcessor final : public KeeperStorageRequestProcessor
-{
- using KeeperStorageRequestProcessor::KeeperStorageRequestProcessor;
-
- KeeperStorage::ResponsesForSessions processWatches(KeeperStorage::Watches & watches, KeeperStorage::Watches & list_watches) const override
- {
- return processWatchesImpl(zk_request->getPath(), watches, list_watches, Coordination::Event::CREATED);
- }
-
- bool checkAuth(KeeperStorage & storage, int64_t session_id) const override
- {
- auto & container = storage.container;
- auto path = zk_request->getPath();
- auto parent_path = parentPath(path);
-
- auto it = container.find(parent_path);
- if (it == container.end())
- return true;
-
- const auto & node_acls = storage.acl_map.convertNumber(it->value.acl_id);
- if (node_acls.empty())
- return true;
-
- const auto & session_auths = storage.session_and_auth[session_id];
- return checkACL(Coordination::ACL::Create, node_acls, session_auths);
- }
-
- std::pair process(KeeperStorage & storage, int64_t zxid, int64_t session_id, int64_t time) const override
- {
- auto & container = storage.container;
- auto & ephemerals = storage.ephemerals;
-
- Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse();
- Undo undo;
- Coordination::ZooKeeperCreateResponse & response = dynamic_cast(*response_ptr);
- Coordination::ZooKeeperCreateRequest & request = dynamic_cast(*zk_request);
-
- auto parent_path = parentPath(request.path);
- auto it = container.find(parent_path);
-
- if (it == container.end())
- {
- response.error = Coordination::Error::ZNONODE;
- return { response_ptr, undo };
- }
- else if (it->value.stat.ephemeralOwner != 0)
- {
- response.error = Coordination::Error::ZNOCHILDRENFOREPHEMERALS;
- return { response_ptr, undo };
- }
- std::string path_created = request.path;
- if (request.is_sequential)
- {
- auto seq_num = it->value.seq_num;
-
- std::stringstream seq_num_str; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
- seq_num_str.exceptions(std::ios::failbit);
- seq_num_str << std::setw(10) << std::setfill('0') << seq_num;
-
- path_created += seq_num_str.str();
- }
- if (container.contains(path_created))
- {
- response.error = Coordination::Error::ZNODEEXISTS;
- return { response_ptr, undo };
- }
- if (getBaseName(path_created).size == 0)
- {
- response.error = Coordination::Error::ZBADARGUMENTS;
- return { response_ptr, undo };
- }
-
- auto & session_auth_ids = storage.session_and_auth[session_id];
-
- KeeperStorage::Node created_node;
-
- Coordination::ACLs node_acls;
- if (!fixupACL(request.acls, session_auth_ids, node_acls))
- {
- response.error = Coordination::Error::ZINVALIDACL;
- return {response_ptr, {}};
- }
-
- uint64_t acl_id = storage.acl_map.convertACLs(node_acls);
- storage.acl_map.addUsage(acl_id);
-
- created_node.acl_id = acl_id;
- created_node.stat.czxid = zxid;
- created_node.stat.mzxid = zxid;
- created_node.stat.pzxid = zxid;
- created_node.stat.ctime = time;
- created_node.stat.mtime = time;
- created_node.stat.numChildren = 0;
- created_node.stat.dataLength = request.data.length();
- created_node.stat.ephemeralOwner = request.is_ephemeral ? session_id : 0;
- created_node.is_sequental = request.is_sequential;
- created_node.setData(std::move(request.data));
-
- auto [map_key, _] = container.insert(path_created, created_node);
- /// Take child path from key owned by map.
- auto child_path = getBaseName(map_key->getKey());
-
- int32_t parent_cversion = request.parent_cversion;
- int64_t prev_parent_zxid;
- int32_t prev_parent_cversion;
- container.updateValue(parent_path, [child_path, zxid, &prev_parent_zxid,
- parent_cversion, &prev_parent_cversion] (KeeperStorage::Node & parent)
- {
- parent.addChild(child_path);
- prev_parent_cversion = parent.stat.cversion;
- prev_parent_zxid = parent.stat.pzxid;
-
- /// Increment sequential number even if node is not sequential
- ++parent.seq_num;
-
- if (parent_cversion == -1)
- ++parent.stat.cversion;
- else if (parent_cversion > parent.stat.cversion)
- parent.stat.cversion = parent_cversion;
-
- if (zxid > parent.stat.pzxid)
- parent.stat.pzxid = zxid;
- ++parent.stat.numChildren;
- });
-
- response.path_created = path_created;
-
- if (request.is_ephemeral)
- ephemerals[session_id].emplace(path_created);
-
- undo = [&storage, prev_parent_zxid, prev_parent_cversion, session_id, path_created, is_ephemeral = request.is_ephemeral, parent_path, child_path, acl_id]
- {
- storage.acl_map.removeUsage(acl_id);
-
- if (is_ephemeral)
- storage.ephemerals[session_id].erase(path_created);
-
- storage.container.updateValue(parent_path, [child_path, prev_parent_zxid, prev_parent_cversion] (KeeperStorage::Node & undo_parent)
- {
- --undo_parent.stat.numChildren;
- --undo_parent.seq_num;
- undo_parent.stat.cversion = prev_parent_cversion;
- undo_parent.stat.pzxid = prev_parent_zxid;
- undo_parent.removeChild(child_path);
- });
-
- storage.container.erase(path_created);
- };
-
- response.error = Coordination::Error::ZOK;
- return { response_ptr, undo };
- }
-};
-
-struct KeeperStorageGetRequestProcessor final : public KeeperStorageRequestProcessor
-{
-
- bool checkAuth(KeeperStorage & storage, int64_t session_id) const override
- {
- auto & container = storage.container;
- auto it = container.find(zk_request->getPath());
- if (it == container.end())
- return true;
-
- const auto & node_acls = storage.acl_map.convertNumber(it->value.acl_id);
- if (node_acls.empty())
- return true;
-
- const auto & session_auths = storage.session_and_auth[session_id];
- return checkACL(Coordination::ACL::Read, node_acls, session_auths);
- }
-
- using KeeperStorageRequestProcessor::KeeperStorageRequestProcessor;
- std::pair process(KeeperStorage & storage, int64_t /* zxid */, int64_t /* session_id */, int64_t /* time */) const override
- {
- auto & container = storage.container;
- Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse();
- Coordination::ZooKeeperGetResponse & response = dynamic_cast(*response_ptr);
- Coordination::ZooKeeperGetRequest & request = dynamic_cast(*zk_request);
-
- auto it = container.find(request.path);
- if (it == container.end())
- {
- response.error = Coordination::Error::ZNONODE;
- }
- else
- {
- response.stat = it->value.stat;
- response.data = it->value.getData();
- response.error = Coordination::Error::ZOK;
- }
-
- return { response_ptr, {} };
+ return response;
}
};
namespace
{
- /// Garbage required to apply log to "fuzzy" zookeeper snapshot
- void updateParentPzxid(const std::string & child_path, int64_t zxid, KeeperStorage::Container & container)
+
+ Coordination::ACLs getNodeACLs(KeeperStorage & storage, StringRef path, bool is_local)
{
- auto parent_path = parentPath(child_path);
- auto parent_it = container.find(parent_path);
- if (parent_it != container.end())
+ if (is_local)
{
- container.updateValue(parent_path, [zxid](KeeperStorage::Node & parent)
- {
- if (parent.stat.pzxid < zxid)
- parent.stat.pzxid = zxid;
- });
+ auto node_it = storage.container.find(path);
+ if (node_it == storage.container.end())
+ return {};
+
+ return storage.acl_map.convertNumber(node_it->value.acl_id);
+ }
+
+ return storage.uncommitted_state.getACLs(path);
+ }
+
+}
+bool KeeperStorage::checkACL(StringRef path, int32_t permission, int64_t session_id, bool is_local)
+{
+ const auto node_acls = getNodeACLs(*this, path, is_local);
+ if (node_acls.empty())
+ return true;
+
+ if (uncommitted_state.hasACL(session_id, is_local, [](const auto & auth_id) { return auth_id.scheme == "super"; }))
+ return true;
+
+
+ for (const auto & node_acl : node_acls)
+ {
+ if (node_acl.permissions & permission)
+ {
+ if (node_acl.scheme == "world" && node_acl.id == "anyone")
+ return true;
+
+ if (uncommitted_state.hasACL(
+ session_id,
+ is_local,
+ [&](const auto & auth_id) { return auth_id.scheme == node_acl.scheme && auth_id.id == node_acl.id; }))
+ return true;
}
}
+
+ return false;
}
-struct KeeperStorageRemoveRequestProcessor final : public KeeperStorageRequestProcessor
+
+struct KeeperStorageCreateRequestProcessor final : public KeeperStorageRequestProcessor
{
- bool checkAuth(KeeperStorage & storage, int64_t session_id) const override
+ using KeeperStorageRequestProcessor::KeeperStorageRequestProcessor;
+
+ KeeperStorage::ResponsesForSessions
+ processWatches(KeeperStorage::Watches & watches, KeeperStorage::Watches & list_watches) const override
{
- auto & container = storage.container;
- auto it = container.find(parentPath(zk_request->getPath()));
- if (it == container.end())
- return true;
+ return processWatchesImpl(zk_request->getPath(), watches, list_watches, Coordination::Event::CREATED);
+ }
- const auto & node_acls = storage.acl_map.convertNumber(it->value.acl_id);
- if (node_acls.empty())
- return true;
+ bool checkAuth(KeeperStorage & storage, int64_t session_id, bool is_local) const override
+ {
+ auto path = zk_request->getPath();
+ return storage.checkACL(parentPath(path), Coordination::ACL::Create, session_id, is_local);
+ }
- const auto & session_auths = storage.session_and_auth[session_id];
- return checkACL(Coordination::ACL::Delete, node_acls, session_auths);
+ std::vector preprocess(KeeperStorage & storage, int64_t zxid, int64_t session_id, int64_t time) const override
+ {
+ Coordination::ZooKeeperCreateRequest & request = dynamic_cast(*zk_request);
+
+ std::vector new_deltas;
+
+ auto parent_path = parentPath(request.path);
+ auto parent_node = storage.uncommitted_state.getNode(parent_path);
+ if (parent_node == nullptr)
+ return {{zxid, Coordination::Error::ZNONODE}};
+
+ else if (parent_node->stat.ephemeralOwner != 0)
+ return {{zxid, Coordination::Error::ZNOCHILDRENFOREPHEMERALS}};
+
+ std::string path_created = request.path;
+ if (request.is_sequential)
+ {
+ auto seq_num = parent_node->seq_num;
+
+ std::stringstream seq_num_str; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
+ seq_num_str.exceptions(std::ios::failbit);
+ seq_num_str << std::setw(10) << std::setfill('0') << seq_num;
+
+ path_created += seq_num_str.str();
+ }
+
+ if (storage.uncommitted_state.hasNode(path_created))
+ return {{zxid, Coordination::Error::ZNODEEXISTS}};
+
+ if (getBaseName(path_created).size == 0)
+ return {{zxid, Coordination::Error::ZBADARGUMENTS}};
+
+ Coordination::ACLs node_acls;
+ if (!fixupACL(request.acls, storage.session_and_auth[session_id], node_acls))
+ return {{zxid, Coordination::Error::ZINVALIDACL}};
+
+ Coordination::Stat stat;
+ stat.czxid = zxid;
+ stat.mzxid = zxid;
+ stat.pzxid = zxid;
+ stat.ctime = time;
+ stat.mtime = time;
+ stat.numChildren = 0;
+ stat.version = 0;
+ stat.aversion = 0;
+ stat.cversion = 0;
+ stat.dataLength = request.data.length();
+ stat.ephemeralOwner = request.is_ephemeral ? session_id : 0;
+
+ new_deltas.emplace_back(
+ std::move(path_created),
+ zxid,
+ KeeperStorage::CreateNodeDelta{stat, request.is_ephemeral, request.is_sequential, std::move(node_acls), request.data});
+
+ int32_t parent_cversion = request.parent_cversion;
+
+ new_deltas.emplace_back(
+ std::string{parent_path},
+ zxid,
+ KeeperStorage::UpdateNodeDelta{[parent_cversion, zxid](KeeperStorage::Node & node)
+ {
+ ++node.seq_num;
+ if (parent_cversion == -1)
+ ++node.stat.cversion;
+ else if (parent_cversion > node.stat.cversion)
+ node.stat.cversion = parent_cversion;
+
+ if (zxid > node.stat.pzxid)
+ node.stat.pzxid = zxid;
+ ++node.stat.numChildren;
+ }});
+ return new_deltas;
+ }
+
+ Coordination::ZooKeeperResponsePtr process(KeeperStorage & storage, int64_t zxid, int64_t session_id, int64_t /*time*/) const override
+ {
+ Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse();
+ Coordination::ZooKeeperCreateResponse & response = dynamic_cast(*response_ptr);
+
+ if (const auto result = storage.commit(zxid, session_id); result != Coordination::Error::ZOK)
+ {
+ response.error = result;
+ return response_ptr;
+ }
+
+ const auto & deltas = storage.uncommitted_state.deltas;
+ auto create_delta_it = std::find_if(
+ deltas.begin(),
+ deltas.end(),
+ [zxid](const auto & delta)
+ { return delta.zxid == zxid && std::holds_alternative(delta.operation); });
+
+ assert(create_delta_it != deltas.end());
+
+ response.path_created = create_delta_it->path;
+ response.error = Coordination::Error::ZOK;
+ return response_ptr;
+ }
+};
+
+struct KeeperStorageGetRequestProcessor final : public KeeperStorageRequestProcessor
+{
+ bool checkAuth(KeeperStorage & storage, int64_t session_id, bool is_local) const override
+ {
+ return storage.checkACL(zk_request->getPath(), Coordination::ACL::Read, session_id, is_local);
}
using KeeperStorageRequestProcessor::KeeperStorageRequestProcessor;
- std::pair process(KeeperStorage & storage, int64_t zxid, int64_t /*session_id*/, int64_t /* time */) const override
+
+ std::vector
+ preprocess(KeeperStorage & storage, int64_t zxid, int64_t /*session_id*/, int64_t /*time*/) const override
{
- auto & container = storage.container;
- auto & ephemerals = storage.ephemerals;
+ Coordination::ZooKeeperGetRequest & request = dynamic_cast(*zk_request);
+ if (!storage.uncommitted_state.hasNode(request.path))
+ return {{zxid, Coordination::Error::ZNONODE}};
+
+ return {};
+ }
+
+ template
+ Coordination::ZooKeeperResponsePtr processImpl(KeeperStorage & storage, int64_t zxid, int64_t session_id, int64_t /* time */) const
+ {
Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse();
- Coordination::ZooKeeperRemoveResponse & response = dynamic_cast(*response_ptr);
- Coordination::ZooKeeperRemoveRequest & request = dynamic_cast(*zk_request);
- Undo undo;
+ Coordination::ZooKeeperGetResponse & response = dynamic_cast(*response_ptr);
+ Coordination::ZooKeeperGetRequest & request = dynamic_cast(*zk_request);
- auto it = container.find(request.path);
- if (it == container.end())
+ if constexpr (!local)
{
- if (request.restored_from_zookeeper_log)
- updateParentPzxid(request.path, zxid, container);
- response.error = Coordination::Error::ZNONODE;
+ if (const auto result = storage.commit(zxid, session_id); result != Coordination::Error::ZOK)
+ {
+ response.error = result;
+ return response_ptr;
+ }
}
- else if (request.version != -1 && request.version != it->value.stat.version)
+
+ auto & container = storage.container;
+ auto node_it = container.find(request.path);
+ if (node_it == container.end())
{
- response.error = Coordination::Error::ZBADVERSION;
- }
- else if (it->value.stat.numChildren)
- {
- response.error = Coordination::Error::ZNOTEMPTY;
+ if constexpr (local)
+ response.error = Coordination::Error::ZNONODE;
+ else
+ onStorageInconsistency();
}
else
{
- if (request.restored_from_zookeeper_log)
- updateParentPzxid(request.path, zxid, container);
-
- auto prev_node = it->value;
- if (prev_node.stat.ephemeralOwner != 0)
- {
- auto ephemerals_it = ephemerals.find(prev_node.stat.ephemeralOwner);
- ephemerals_it->second.erase(request.path);
- if (ephemerals_it->second.empty())
- ephemerals.erase(ephemerals_it);
- }
-
- storage.acl_map.removeUsage(prev_node.acl_id);
-
- container.updateValue(parentPath(request.path), [child_basename = getBaseName(it->key)] (KeeperStorage::Node & parent)
- {
- --parent.stat.numChildren;
- ++parent.stat.cversion;
- parent.removeChild(child_basename);
- });
-
+ response.stat = node_it->value.stat;
+ response.data = node_it->value.getData();
response.error = Coordination::Error::ZOK;
- /// Erase full path from container after child removed from parent
- container.erase(request.path);
-
- undo = [prev_node, &storage, path = request.path]
- {
- if (prev_node.stat.ephemeralOwner != 0)
- storage.ephemerals[prev_node.stat.ephemeralOwner].emplace(path);
-
- storage.acl_map.addUsage(prev_node.acl_id);
-
- /// Dangerous place: we are adding StringRef to child into children unordered_hash set.
- /// That's why we are taking getBaseName from inserted key, not from the path from request object.
- auto [map_key, _] = storage.container.insert(path, prev_node);
- storage.container.updateValue(parentPath(path), [child_name = getBaseName(map_key->getKey())] (KeeperStorage::Node & parent)
- {
- ++parent.stat.numChildren;
- --parent.stat.cversion;
- parent.addChild(child_name);
- });
- };
}
- return { response_ptr, undo };
+ return response_ptr;
}
- KeeperStorage::ResponsesForSessions processWatches(KeeperStorage::Watches & watches, KeeperStorage::Watches & list_watches) const override
+
+ Coordination::ZooKeeperResponsePtr process(KeeperStorage & storage, int64_t zxid, int64_t session_id, int64_t time) const override
+ {
+ return processImpl(storage, zxid, session_id, time);
+ }
+
+ Coordination::ZooKeeperResponsePtr processLocal(KeeperStorage & storage, int64_t zxid, int64_t session_id, int64_t time) const override
+ {
+ return processImpl(storage, zxid, session_id, time);
+ }
+};
+
+struct KeeperStorageRemoveRequestProcessor final : public KeeperStorageRequestProcessor
+{
+ bool checkAuth(KeeperStorage & storage, int64_t session_id, bool is_local) const override
+ {
+ return storage.checkACL(parentPath(zk_request->getPath()), Coordination::ACL::Delete, session_id, is_local);
+ }
+
+ using KeeperStorageRequestProcessor::KeeperStorageRequestProcessor;
+ std::vector
+ preprocess(KeeperStorage & storage, int64_t zxid, int64_t /*session_id*/, int64_t /*time*/) const override
+ {
+ Coordination::ZooKeeperRemoveRequest & request = dynamic_cast(*zk_request);
+
+ std::vector new_deltas;
+
+ const auto update_parent_pzxid = [&]()
+ {
+ auto parent_path = parentPath(request.path);
+ if (!storage.uncommitted_state.hasNode(parent_path))
+ return;
+
+ new_deltas.emplace_back(
+ std::string{parent_path},
+ zxid,
+ KeeperStorage::UpdateNodeDelta{[zxid](KeeperStorage::Node & parent)
+ {
+ if (parent.stat.pzxid < zxid)
+ parent.stat.pzxid = zxid;
+ }});
+ };
+
+ auto node = storage.uncommitted_state.getNode(request.path);
+
+ if (!node)
+ {
+ if (request.restored_from_zookeeper_log)
+ update_parent_pzxid();
+ return {{zxid, Coordination::Error::ZNONODE}};
+ }
+ else if (request.version != -1 && request.version != node->stat.version)
+ return {{zxid, Coordination::Error::ZBADVERSION}};
+ else if (node->stat.numChildren)
+ return {{zxid, Coordination::Error::ZNOTEMPTY}};
+
+ if (request.restored_from_zookeeper_log)
+ update_parent_pzxid();
+
+ new_deltas.emplace_back(
+ std::string{parentPath(request.path)},
+ zxid,
+ KeeperStorage::UpdateNodeDelta{[](KeeperStorage::Node & parent)
+ {
+ --parent.stat.numChildren;
+ ++parent.stat.cversion;
+ }});
+
+ new_deltas.emplace_back(request.path, zxid, KeeperStorage::RemoveNodeDelta{request.version});
+
+ return new_deltas;
+ }
+
+ Coordination::ZooKeeperResponsePtr process(KeeperStorage & storage, int64_t zxid, int64_t session_id, int64_t /* time */) const override
+ {
+ Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse();
+ Coordination::ZooKeeperRemoveResponse & response = dynamic_cast(*response_ptr);
+
+ response.error = storage.commit(zxid, session_id);
+ return response_ptr;
+ }
+
+ KeeperStorage::ResponsesForSessions
+ processWatches(KeeperStorage::Watches & watches, KeeperStorage::Watches & list_watches) const override
{
return processWatchesImpl(zk_request->getPath(), watches, list_watches, Coordination::Event::DELETED);
}
@@ -556,101 +829,140 @@ struct KeeperStorageRemoveRequestProcessor final : public KeeperStorageRequestPr
struct KeeperStorageExistsRequestProcessor final : public KeeperStorageRequestProcessor
{
using KeeperStorageRequestProcessor::KeeperStorageRequestProcessor;
- std::pair process(KeeperStorage & storage, int64_t /*zxid*/, int64_t /* session_id */, int64_t /* time */) const override
- {
- auto & container = storage.container;
+ std::vector
+ preprocess(KeeperStorage & storage, int64_t zxid, int64_t /*session_id*/, int64_t /*time*/) const override
+ {
+ Coordination::ZooKeeperExistsRequest & request = dynamic_cast(*zk_request);
+
+ if (!storage.uncommitted_state.hasNode(request.path))
+ return {{zxid, Coordination::Error::ZNONODE}};
+
+ return {};
+ }
+
+ template
+ Coordination::ZooKeeperResponsePtr processImpl(KeeperStorage & storage, int64_t zxid, int64_t session_id, int64_t /* time */) const
+ {
Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse();
Coordination::ZooKeeperExistsResponse & response = dynamic_cast(*response_ptr);
Coordination::ZooKeeperExistsRequest & request = dynamic_cast(*zk_request);
- auto it = container.find(request.path);
- if (it != container.end())
+ if constexpr (!local)
{
- response.stat = it->value.stat;
- response.error = Coordination::Error::ZOK;
+ if (const auto result = storage.commit(zxid, session_id); result != Coordination::Error::ZOK)
+ {
+ response.error = result;
+ return response_ptr;
+ }
+ }
+
+ auto & container = storage.container;
+ auto node_it = container.find(request.path);
+ if (node_it == container.end())
+ {
+ if constexpr (local)
+ response.error = Coordination::Error::ZNONODE;
+ else
+ onStorageInconsistency();
}
else
{
- response.error = Coordination::Error::ZNONODE;
+ response.stat = node_it->value.stat;
+ response.error = Coordination::Error::ZOK;
}
- return { response_ptr, {} };
+ return response_ptr;
+ }
+
+ Coordination::ZooKeeperResponsePtr process(KeeperStorage & storage, int64_t zxid, int64_t session_id, int64_t time) const override
+ {
+ return processImpl(storage, zxid, session_id, time);
+ }
+
+ Coordination::ZooKeeperResponsePtr processLocal(KeeperStorage & storage, int64_t zxid, int64_t session_id, int64_t time) const override
+ {
+ return processImpl(storage, zxid, session_id, time);
}
};
struct KeeperStorageSetRequestProcessor final : public KeeperStorageRequestProcessor
{
- bool checkAuth(KeeperStorage & storage, int64_t session_id) const override
+ bool checkAuth(KeeperStorage & storage, int64_t session_id, bool is_local) const override
{
- auto & container = storage.container;
- auto it = container.find(zk_request->getPath());
- if (it == container.end())
- return true;
-
- const auto & node_acls = storage.acl_map.convertNumber(it->value.acl_id);
- if (node_acls.empty())
- return true;
-
- const auto & session_auths = storage.session_and_auth[session_id];
- return checkACL(Coordination::ACL::Write, node_acls, session_auths);
+ return storage.checkACL(zk_request->getPath(), Coordination::ACL::Write, session_id, is_local);
}
using KeeperStorageRequestProcessor::KeeperStorageRequestProcessor;
- std::pair process(KeeperStorage & storage, int64_t zxid, int64_t /* session_id */, int64_t time) const override
+ std::vector preprocess(KeeperStorage & storage, int64_t zxid, int64_t /*session_id*/, int64_t time) const override
+ {
+ Coordination::ZooKeeperSetRequest & request = dynamic_cast(*zk_request);
+
+ std::vector new_deltas;
+
+ if (!storage.uncommitted_state.hasNode(request.path))
+ return {{zxid, Coordination::Error::ZNONODE}};
+
+ auto node = storage.uncommitted_state.getNode(request.path);
+
+ if (request.version != -1 && request.version != node->stat.version)
+ return {{zxid, Coordination::Error::ZBADVERSION}};
+
+ new_deltas.emplace_back(
+ request.path,
+ zxid,
+ KeeperStorage::UpdateNodeDelta{
+ [zxid, data = request.data, time](KeeperStorage::Node & value)
+ {
+ value.stat.version++;
+ value.stat.mzxid = zxid;
+ value.stat.mtime = time;
+ value.stat.dataLength = data.length();
+ value.setData(data);
+ },
+ request.version});
+
+ new_deltas.emplace_back(
+ parentPath(request.path).toString(),
+ zxid,
+ KeeperStorage::UpdateNodeDelta
+ {
+ [](KeeperStorage::Node & parent)
+ {
+ parent.stat.cversion++;
+ }
+ }
+ );
+
+ return new_deltas;
+ }
+
+ Coordination::ZooKeeperResponsePtr process(KeeperStorage & storage, int64_t zxid, int64_t session_id, int64_t /*time*/) const override
{
auto & container = storage.container;
Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse();
Coordination::ZooKeeperSetResponse & response = dynamic_cast(*response_ptr);
Coordination::ZooKeeperSetRequest & request = dynamic_cast(*zk_request);
- Undo undo;
- auto it = container.find(request.path);
- if (it == container.end())
+ if (const auto result = storage.commit(zxid, session_id); result != Coordination::Error::ZOK)
{
- response.error = Coordination::Error::ZNONODE;
- }
- else if (request.version == -1 || request.version == it->value.stat.version)
- {
-
- auto prev_node = it->value;
-
- auto itr = container.updateValue(request.path, [zxid, request, time] (KeeperStorage::Node & value) mutable
- {
- value.stat.version++;
- value.stat.mzxid = zxid;
- value.stat.mtime = time;
- value.stat.dataLength = request.data.length();
- value.setData(std::move(request.data));
- });
-
- container.updateValue(parentPath(request.path), [] (KeeperStorage::Node & parent)
- {
- parent.stat.cversion++;
- });
-
- response.stat = itr->value.stat;
- response.error = Coordination::Error::ZOK;
-
- undo = [prev_node, &container, path = request.path]
- {
- container.updateValue(path, [&prev_node] (KeeperStorage::Node & value) { value = prev_node; });
- container.updateValue(parentPath(path), [] (KeeperStorage::Node & parent)
- {
- parent.stat.cversion--;
- });
- };
- }
- else
- {
- response.error = Coordination::Error::ZBADVERSION;
+ response.error = result;
+ return response_ptr;
}
- return { response_ptr, undo };
+ auto node_it = container.find(request.path);
+ if (node_it == container.end())
+ onStorageInconsistency();
+
+ response.stat = node_it->value.stat;
+ response.error = Coordination::Error::ZOK;
+
+ return response_ptr;
}
- KeeperStorage::ResponsesForSessions processWatches(KeeperStorage::Watches & watches, KeeperStorage::Watches & list_watches) const override
+ KeeperStorage::ResponsesForSessions
+ processWatches(KeeperStorage::Watches & watches, KeeperStorage::Watches & list_watches) const override
{
return processWatchesImpl(zk_request->getPath(), watches, list_watches, Coordination::Event::CHANGED);
}
@@ -658,33 +970,48 @@ struct KeeperStorageSetRequestProcessor final : public KeeperStorageRequestProce
struct KeeperStorageListRequestProcessor final : public KeeperStorageRequestProcessor
{
- bool checkAuth(KeeperStorage & storage, int64_t session_id) const override
+ bool checkAuth(KeeperStorage & storage, int64_t session_id, bool is_local) const override
{
- auto & container = storage.container;
- auto it = container.find(zk_request->getPath());
- if (it == container.end())
- return true;
-
- const auto & node_acls = storage.acl_map.convertNumber(it->value.acl_id);
- if (node_acls.empty())
- return true;
-
- const auto & session_auths = storage.session_and_auth[session_id];
- return checkACL(Coordination::ACL::Read, node_acls, session_auths);
+ return storage.checkACL(zk_request->getPath(), Coordination::ACL::Read, session_id, is_local);
}
using KeeperStorageRequestProcessor::KeeperStorageRequestProcessor;
- std::pair process(KeeperStorage & storage, int64_t /*zxid*/, int64_t /*session_id*/, int64_t /* time */) const override
+ std::vector
+ preprocess(KeeperStorage & storage, int64_t zxid, int64_t /*session_id*/, int64_t /*time*/) const override
+ {
+ Coordination::ZooKeeperListRequest & request = dynamic_cast(*zk_request);
+
+ if (!storage.uncommitted_state.hasNode(request.path))
+ return {{zxid, Coordination::Error::ZNONODE}};
+
+ return {};
+ }
+
+
+ template
+ Coordination::ZooKeeperResponsePtr processImpl(KeeperStorage & storage, int64_t zxid, int64_t session_id, int64_t /* time */) const
{
- auto & container = storage.container;
Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse();
Coordination::ZooKeeperListResponse & response = dynamic_cast(*response_ptr);
Coordination::ZooKeeperListRequest & request = dynamic_cast(*zk_request);
- auto it = container.find(request.path);
- if (it == container.end())
+ if constexpr (!local)
{
- response.error = Coordination::Error::ZNONODE;
+ if (const auto result = storage.commit(zxid, session_id); result != Coordination::Error::ZOK)
+ {
+ response.error = result;
+ return response_ptr;
+ }
+ }
+
+ auto & container = storage.container;
+ auto node_it = container.find(request.path);
+ if (node_it == container.end())
+ {
+ if constexpr (local)
+ response.error = Coordination::Error::ZNONODE;
+ else
+ onStorageInconsistency();
}
else
{
@@ -692,174 +1019,247 @@ struct KeeperStorageListRequestProcessor final : public KeeperStorageRequestProc
if (path_prefix.empty())
throw DB::Exception("Logical error: path cannot be empty", ErrorCodes::LOGICAL_ERROR);
- const auto & children = it->value.getChildren();
+ const auto & children = node_it->value.getChildren();
response.names.reserve(children.size());
for (const auto child : children)
response.names.push_back(child.toString());
- response.stat = it->value.stat;
+ response.stat = node_it->value.stat;
response.error = Coordination::Error::ZOK;
}
- return { response_ptr, {} };
+ return response_ptr;
+ }
+
+ Coordination::ZooKeeperResponsePtr process(KeeperStorage & storage, int64_t zxid, int64_t session_id, int64_t time) const override
+ {
+ return processImpl(storage, zxid, session_id, time);
+ }
+
+ Coordination::ZooKeeperResponsePtr processLocal(KeeperStorage & storage, int64_t zxid, int64_t session_id, int64_t time) const override
+ {
+ return processImpl(storage, zxid, session_id, time);
}
};
struct KeeperStorageCheckRequestProcessor final : public KeeperStorageRequestProcessor
{
- bool checkAuth(KeeperStorage & storage, int64_t session_id) const override
+ bool checkAuth(KeeperStorage & storage, int64_t session_id, bool is_local) const override
{
- auto & container = storage.container;
- auto it = container.find(zk_request->getPath());
- if (it == container.end())
- return true;
-
- const auto & node_acls = storage.acl_map.convertNumber(it->value.acl_id);
- if (node_acls.empty())
- return true;
-
- const auto & session_auths = storage.session_and_auth[session_id];
- return checkACL(Coordination::ACL::Read, node_acls, session_auths);
+ return storage.checkACL(zk_request->getPath(), Coordination::ACL::Read, session_id, is_local);
}
using KeeperStorageRequestProcessor::KeeperStorageRequestProcessor;
- std::pair process(KeeperStorage & storage, int64_t /*zxid*/, int64_t /*session_id*/, int64_t /* time */) const override
+ std::vector
+ preprocess(KeeperStorage & storage, int64_t zxid, int64_t /*session_id*/, int64_t /*time*/) const override
{
- auto & container = storage.container;
+ Coordination::ZooKeeperCheckRequest & request = dynamic_cast(*zk_request);
+ if (!storage.uncommitted_state.hasNode(request.path))
+ return {{zxid, Coordination::Error::ZNONODE}};
+
+ auto node = storage.uncommitted_state.getNode(request.path);
+ if (request.version != -1 && request.version != node->stat.version)
+ return {{zxid, Coordination::Error::ZBADVERSION}};
+
+ return {};
+ }
+
+ template
+ Coordination::ZooKeeperResponsePtr processImpl(KeeperStorage & storage, int64_t zxid, int64_t session_id, int64_t /* time */) const
+ {
Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse();
Coordination::ZooKeeperCheckResponse & response = dynamic_cast(*response_ptr);
Coordination::ZooKeeperCheckRequest & request = dynamic_cast(*zk_request);
- auto it = container.find(request.path);
- if (it == container.end())
+
+ if constexpr (!local)
{
- response.error = Coordination::Error::ZNONODE;
+ if (const auto result = storage.commit(zxid, session_id); result != Coordination::Error::ZOK)
+ {
+ response.error = result;
+ return response_ptr;
+ }
}
- else if (request.version != -1 && request.version != it->value.stat.version)
+
+ const auto on_error = [&]([[maybe_unused]] const auto error_code)
{
- response.error = Coordination::Error::ZBADVERSION;
+ if constexpr (local)
+ response.error = error_code;
+ else
+ onStorageInconsistency();
+ };
+
+ auto & container = storage.container;
+ auto node_it = container.find(request.path);
+ if (node_it == container.end())
+ {
+ on_error(Coordination::Error::ZNONODE);
+ }
+ else if (request.version != -1 && request.version != node_it->value.stat.version)
+ {
+ on_error(Coordination::Error::ZBADVERSION);
}
else
{
response.error = Coordination::Error::ZOK;
}
- return { response_ptr, {} };
+ return response_ptr;
+ }
+
+ Coordination::ZooKeeperResponsePtr process(KeeperStorage & storage, int64_t zxid, int64_t session_id, int64_t time) const override
+ {
+ return processImpl(storage, zxid, session_id, time);
+ }
+
+ Coordination::ZooKeeperResponsePtr processLocal(KeeperStorage & storage, int64_t zxid, int64_t session_id, int64_t time) const override
+ {
+ return processImpl(storage, zxid, session_id, time);
}
};
struct KeeperStorageSetACLRequestProcessor final : public KeeperStorageRequestProcessor
{
- bool checkAuth(KeeperStorage & storage, int64_t session_id) const override
+ bool checkAuth(KeeperStorage & storage, int64_t session_id, bool is_local) const override
{
- auto & container = storage.container;
- auto it = container.find(zk_request->getPath());
- if (it == container.end())
- return true;
-
- const auto & node_acls = storage.acl_map.convertNumber(it->value.acl_id);
- if (node_acls.empty())
- return true;
-
- const auto & session_auths = storage.session_and_auth[session_id];
- return checkACL(Coordination::ACL::Admin, node_acls, session_auths);
+ return storage.checkACL(zk_request->getPath(), Coordination::ACL::Admin, session_id, is_local);
}
using KeeperStorageRequestProcessor::KeeperStorageRequestProcessor;
- std::pair process(KeeperStorage & storage, int64_t /*zxid*/, int64_t session_id, int64_t /* time */) const override
+ std::vector preprocess(KeeperStorage & storage, int64_t zxid, int64_t session_id, int64_t /*time*/) const override
{
- auto & container = storage.container;
+ Coordination::ZooKeeperSetACLRequest & request = dynamic_cast(*zk_request);
+ auto & uncommitted_state = storage.uncommitted_state;
+ if (!uncommitted_state.hasNode(request.path))
+ return {{zxid, Coordination::Error::ZNONODE}};
+
+ auto node = uncommitted_state.getNode(request.path);
+
+ if (request.version != -1 && request.version != node->stat.aversion)
+ return {{zxid, Coordination::Error::ZBADVERSION}};
+
+
+ auto & session_auth_ids = storage.session_and_auth[session_id];
+ Coordination::ACLs node_acls;
+
+ if (!fixupACL(request.acls, session_auth_ids, node_acls))
+ return {{zxid, Coordination::Error::ZINVALIDACL}};
+
+ return
+ {
+ {
+ request.path,
+ zxid,
+ KeeperStorage::SetACLDelta{std::move(node_acls), request.version}
+ },
+ {
+ request.path,
+ zxid,
+ KeeperStorage::UpdateNodeDelta
+ {
+ [](KeeperStorage::Node & n) { ++n.stat.aversion; }
+ }
+ }
+ };
+ }
+
+ Coordination::ZooKeeperResponsePtr process(KeeperStorage & storage, int64_t zxid, int64_t session_id, int64_t /* time */) const override
+ {
Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse();
Coordination::ZooKeeperSetACLResponse & response = dynamic_cast(*response_ptr);
Coordination::ZooKeeperSetACLRequest & request = dynamic_cast(*zk_request);
- auto it = container.find(request.path);
- if (it == container.end())
+
+ if (const auto result = storage.commit(zxid, session_id); result != Coordination::Error::ZOK)
{
- response.error = Coordination::Error::ZNONODE;
- }
- else if (request.version != -1 && request.version != it->value.stat.aversion)
- {
- response.error = Coordination::Error::ZBADVERSION;
- }
- else
- {
- auto & session_auth_ids = storage.session_and_auth[session_id];
- Coordination::ACLs node_acls;
-
- if (!fixupACL(request.acls, session_auth_ids, node_acls))
- {
- response.error = Coordination::Error::ZINVALIDACL;
- return {response_ptr, {}};
- }
-
- uint64_t acl_id = storage.acl_map.convertACLs(node_acls);
- storage.acl_map.addUsage(acl_id);
-
- storage.container.updateValue(request.path, [acl_id] (KeeperStorage::Node & node)
- {
- node.acl_id = acl_id;
- ++node.stat.aversion;
- });
-
- response.stat = it->value.stat;
- response.error = Coordination::Error::ZOK;
+ response.error = result;
+ return response_ptr;
}
- /// It cannot be used insied multitransaction?
- return { response_ptr, {} };
+ auto node_it = storage.container.find(request.path);
+ if (node_it == storage.container.end())
+ onStorageInconsistency();
+ response.stat = node_it->value.stat;
+ response.error = Coordination::Error::ZOK;
+
+ return response_ptr;
}
};
struct KeeperStorageGetACLRequestProcessor final : public KeeperStorageRequestProcessor
{
- bool checkAuth(KeeperStorage & storage, int64_t session_id) const override
+ bool checkAuth(KeeperStorage & storage, int64_t session_id, bool is_local) const override
{
- auto & container = storage.container;
- auto it = container.find(zk_request->getPath());
- if (it == container.end())
- return true;
-
- const auto & node_acls = storage.acl_map.convertNumber(it->value.acl_id);
- if (node_acls.empty())
- return true;
-
- const auto & session_auths = storage.session_and_auth[session_id];
- /// LOL, GetACL require more permissions, then SetACL...
- return checkACL(Coordination::ACL::Admin | Coordination::ACL::Read, node_acls, session_auths);
+ return storage.checkACL(zk_request->getPath(), Coordination::ACL::Admin | Coordination::ACL::Read, session_id, is_local);
}
+
using KeeperStorageRequestProcessor::KeeperStorageRequestProcessor;
- std::pair process(KeeperStorage & storage, int64_t /*zxid*/, int64_t /*session_id*/, int64_t /* time */) const override
+ std::vector
+ preprocess(KeeperStorage & storage, int64_t zxid, int64_t /*session_id*/, int64_t /*time*/) const override
+ {
+ Coordination::ZooKeeperGetACLRequest & request = dynamic_cast(*zk_request);
+
+ if (!storage.uncommitted_state.hasNode(request.path))
+ return {{zxid, Coordination::Error::ZNONODE}};
+
+ return {};
+ }
+
+ template
+ Coordination::ZooKeeperResponsePtr processImpl(KeeperStorage & storage, int64_t zxid, int64_t session_id, int64_t /* time */) const
{
Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse();
Coordination::ZooKeeperGetACLResponse & response = dynamic_cast(*response_ptr);
Coordination::ZooKeeperGetACLRequest & request = dynamic_cast(*zk_request);
- auto & container = storage.container;
- auto it = container.find(request.path);
- if (it == container.end())
+
+ if constexpr (!local)
{
- response.error = Coordination::Error::ZNONODE;
+ if (const auto result = storage.commit(zxid, session_id); result != Coordination::Error::ZOK)
+ {
+ response.error = result;
+ return response_ptr;
+ }
+ }
+
+ auto & container = storage.container;
+ auto node_it = container.find(request.path);
+ if (node_it == container.end())
+ {
+ if constexpr (local)
+ response.error = Coordination::Error::ZNONODE;
+ else
+ onStorageInconsistency();
}
else
{
- response.stat = it->value.stat;
- response.acl = storage.acl_map.convertNumber(it->value.acl_id);
+ response.stat = node_it->value.stat;
+ response.acl = storage.acl_map.convertNumber(node_it->value.acl_id);
}
- return {response_ptr, {}};
+ return response_ptr;
+ }
+
+ Coordination::ZooKeeperResponsePtr process(KeeperStorage & storage, int64_t zxid, int64_t session_id, int64_t time) const override
+ {
+ return processImpl(storage, zxid, session_id, time);
+ }
+
+ Coordination::ZooKeeperResponsePtr processLocal(KeeperStorage & storage, int64_t zxid, int64_t session_id, int64_t time) const override
+ {
+ return processImpl(storage, zxid, session_id, time);
}
};
struct KeeperStorageMultiRequestProcessor final : public KeeperStorageRequestProcessor
{
- bool checkAuth(KeeperStorage & storage, int64_t session_id) const override
+ bool checkAuth(KeeperStorage & storage, int64_t session_id, bool is_local) const override
{
for (const auto & concrete_request : concrete_requests)
- if (!concrete_request->checkAuth(storage, session_id))
+ if (!concrete_request->checkAuth(storage, session_id, is_local))
return false;
return true;
}
@@ -889,65 +1289,124 @@ struct KeeperStorageMultiRequestProcessor final : public KeeperStorageRequestPro
concrete_requests.push_back(std::make_shared(sub_zk_request));
break;
default:
- throw DB::Exception(ErrorCodes::BAD_ARGUMENTS, "Illegal command as part of multi ZooKeeper request {}", sub_zk_request->getOpNum());
+ throw DB::Exception(
+ ErrorCodes::BAD_ARGUMENTS, "Illegal command as part of multi ZooKeeper request {}", sub_zk_request->getOpNum());
}
}
}
- std::pair process(KeeperStorage & storage, int64_t zxid, int64_t session_id, int64_t time) const override
+ std::vector preprocess(KeeperStorage & storage, int64_t zxid, int64_t session_id, int64_t time) const override
+ {
+ // manually add deltas so that the result of previous request in the transaction is used in the next request
+ auto & saved_deltas = storage.uncommitted_state.deltas;
+
+ std::vector response_errors;
+ response_errors.reserve(concrete_requests.size());
+ for (size_t i = 0; i < concrete_requests.size(); ++i)
+ {
+ auto new_deltas = concrete_requests[i]->preprocess(storage, zxid, session_id, time);
+
+ if (!new_deltas.empty())
+ {
+ if (auto * error = std::get_if(&new_deltas.back().operation))
+ {
+ std::erase_if(saved_deltas, [zxid](const auto & delta) { return delta.zxid == zxid; });
+
+ response_errors.push_back(error->error);
+
+ for (size_t j = i + 1; j < concrete_requests.size(); ++j)
+ {
+ response_errors.push_back(Coordination::Error::ZRUNTIMEINCONSISTENCY);
+ }
+
+ return {{zxid, KeeperStorage::FailedMultiDelta{std::move(response_errors)}}};
+ }
+ }
+ new_deltas.emplace_back(zxid, KeeperStorage::SubDeltaEnd{});
+ response_errors.push_back(Coordination::Error::ZOK);
+
+ saved_deltas.insert(saved_deltas.end(), std::make_move_iterator(new_deltas.begin()), std::make_move_iterator(new_deltas.end()));
+ }
+
+ return {};
+ }
+
+ Coordination::ZooKeeperResponsePtr process(KeeperStorage & storage, int64_t zxid, int64_t session_id, int64_t time) const override
{
Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse();
Coordination::ZooKeeperMultiResponse & response = dynamic_cast(*response_ptr);
- std::vector undo_actions;
- try
+ auto & deltas = storage.uncommitted_state.deltas;
+ // the deltas will have at least SubDeltaEnd or FailedMultiDelta
+ assert(!deltas.empty());
+ if (auto * failed_multi = std::get_if(&deltas.front().operation))
{
- size_t i = 0;
- for (const auto & concrete_request : concrete_requests)
+ for (size_t i = 0; i < concrete_requests.size(); ++i)
{
- auto [ cur_response, undo_action ] = concrete_request->process(storage, zxid, session_id, time);
-
- response.responses[i] = cur_response;
- if (cur_response->error != Coordination::Error::ZOK)
- {
- for (size_t j = 0; j <= i; ++j)
- {
- auto response_error = response.responses[j]->error;
- response.responses[j] = std::make_shared();
- response.responses[j]->error = response_error;
- }
-
- for (size_t j = i + 1; j < response.responses.size(); ++j)
- {
- response.responses[j] = std::make_shared();
- response.responses[j]->error = Coordination::Error::ZRUNTIMEINCONSISTENCY;
- }
-
- for (auto it = undo_actions.rbegin(); it != undo_actions.rend(); ++it)
- if (*it)
- (*it)();
-
- return { response_ptr, {} };
- }
- else
- undo_actions.emplace_back(std::move(undo_action));
-
- ++i;
+ response.responses[i] = std::make_shared();
+ response.responses[i]->error = failed_multi->error_codes[i];
}
- response.error = Coordination::Error::ZOK;
- return { response_ptr, {} };
+ return response_ptr;
}
- catch (...)
+
+ for (size_t i = 0; i < concrete_requests.size(); ++i)
{
- for (auto it = undo_actions.rbegin(); it != undo_actions.rend(); ++it)
- if (*it)
- (*it)();
- throw;
+ auto cur_response = concrete_requests[i]->process(storage, zxid, session_id, time);
+
+ while (!deltas.empty())
+ {
+ if (std::holds_alternative(deltas.front().operation))
+ {
+ deltas.pop_front();
+ break;
+ }
+
+ deltas.pop_front();
+ }
+
+ response.responses[i] = cur_response;
}
+
+ response.error = Coordination::Error::ZOK;
+ return response_ptr;
}
- KeeperStorage::ResponsesForSessions processWatches(KeeperStorage::Watches & watches, KeeperStorage::Watches & list_watches) const override
+ Coordination::ZooKeeperResponsePtr processLocal(KeeperStorage & storage, int64_t zxid, int64_t session_id, int64_t time) const override
+ {
+ Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse();
+ Coordination::ZooKeeperMultiResponse & response = dynamic_cast(*response_ptr);
+
+ for (size_t i = 0; i < concrete_requests.size(); ++i)
+ {
+ auto cur_response = concrete_requests[i]->process(storage, zxid, session_id, time);
+
+ response.responses[i] = cur_response;
+ if (cur_response->error != Coordination::Error::ZOK)
+ {
+ for (size_t j = 0; j <= i; ++j)
+ {
+ auto response_error = response.responses[j]->error;
+ response.responses[j] = std::make_shared();
+ response.responses[j]->error = response_error;
+ }
+
+ for (size_t j = i + 1; j < response.responses.size(); ++j)
+ {
+ response.responses[j] = std::make_shared();
+ response.responses[j]->error = Coordination::Error::ZRUNTIMEINCONSISTENCY;
+ }
+
+ return response_ptr;
+ }
+ }
+
+ response.error = Coordination::Error::ZOK;
+ return response_ptr;
+ }
+
+ KeeperStorage::ResponsesForSessions
+ processWatches(KeeperStorage::Watches & watches, KeeperStorage::Watches & list_watches) const override
{
KeeperStorage::ResponsesForSessions result;
for (const auto & generic_request : concrete_requests)
@@ -962,7 +1421,7 @@ struct KeeperStorageMultiRequestProcessor final : public KeeperStorageRequestPro
struct KeeperStorageCloseRequestProcessor final : public KeeperStorageRequestProcessor
{
using KeeperStorageRequestProcessor::KeeperStorageRequestProcessor;
- std::pair process(KeeperStorage &, int64_t, int64_t, int64_t /* time */) const override
+ Coordination::ZooKeeperResponsePtr process(KeeperStorage &, int64_t, int64_t, int64_t /* time */) const override
{
throw DB::Exception("Called process on close request", ErrorCodes::LOGICAL_ERROR);
}
@@ -971,36 +1430,40 @@ struct KeeperStorageCloseRequestProcessor final : public KeeperStorageRequestPro
struct KeeperStorageAuthRequestProcessor final : public KeeperStorageRequestProcessor
{
using KeeperStorageRequestProcessor::KeeperStorageRequestProcessor;
- std::pair process(KeeperStorage & storage, int64_t /*zxid*/, int64_t session_id, int64_t /* time */) const override
+ std::vector preprocess(KeeperStorage & storage, int64_t zxid, int64_t session_id, int64_t /*time*/) const override
{
Coordination::ZooKeeperAuthRequest & auth_request = dynamic_cast(*zk_request);
Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse();
- Coordination::ZooKeeperAuthResponse & auth_response = dynamic_cast(*response_ptr);
- auto & sessions_and_auth = storage.session_and_auth;
if (auth_request.scheme != "digest" || std::count(auth_request.data.begin(), auth_request.data.end(), ':') != 1)
+ return {{zxid, Coordination::Error::ZAUTHFAILED}};
+
+ std::vector new_deltas;
+ auto digest = generateDigest(auth_request.data);
+ if (digest == storage.superdigest)
{
- auth_response.error = Coordination::Error::ZAUTHFAILED;
+ KeeperStorage::AuthID auth{"super", ""};
+ new_deltas.emplace_back(zxid, KeeperStorage::AddAuthDelta{session_id, std::move(auth)});
}
else
{
- auto digest = generateDigest(auth_request.data);
- if (digest == storage.superdigest)
- {
- KeeperStorage::AuthID auth{"super", ""};
- sessions_and_auth[session_id].emplace_back(auth);
- }
- else
- {
- KeeperStorage::AuthID auth{auth_request.scheme, digest};
- auto & session_ids = sessions_and_auth[session_id];
- if (std::find(session_ids.begin(), session_ids.end(), auth) == session_ids.end())
- sessions_and_auth[session_id].emplace_back(auth);
- }
-
+ KeeperStorage::AuthID new_auth{auth_request.scheme, digest};
+ if (!storage.uncommitted_state.hasACL(session_id, false, [&](const auto & auth_id) { return new_auth == auth_id; }))
+ new_deltas.emplace_back(zxid, KeeperStorage::AddAuthDelta{session_id, std::move(new_auth)});
}
- return { response_ptr, {} };
+ return new_deltas;
+ }
+
+ Coordination::ZooKeeperResponsePtr process(KeeperStorage & storage, int64_t zxid, int64_t session_id, int64_t /* time */) const override
+ {
+ Coordination::ZooKeeperResponsePtr response_ptr = zk_request->makeResponse();
+ Coordination::ZooKeeperAuthResponse & auth_response = dynamic_cast(*response_ptr);
+
+ if (const auto result = storage.commit(zxid, session_id); result != Coordination::Error::ZOK)
+ auth_response.error = result;
+
+ return response_ptr;
}
};
@@ -1026,7 +1489,6 @@ void KeeperStorage::finalize()
class KeeperStorageRequestProcessorsFactory final : private boost::noncopyable
{
-
public:
using Creator = std::function;
using OpNumToRequest = std::unordered_map;
@@ -1039,11 +1501,11 @@ public:
KeeperStorageRequestProcessorPtr get(const Coordination::ZooKeeperRequestPtr & zk_request) const
{
- auto it = op_num_to_request.find(zk_request->getOpNum());
- if (it == op_num_to_request.end())
+ auto request_it = op_num_to_request.find(zk_request->getOpNum());
+ if (request_it == op_num_to_request.end())
throw DB::Exception("Unknown operation type " + toString(zk_request->getOpNum()), ErrorCodes::LOGICAL_ERROR);
- return it->second(zk_request);
+ return request_it->second(zk_request);
}
void registerRequest(Coordination::OpNum op_num, Creator creator)
@@ -1057,10 +1519,11 @@ private:
KeeperStorageRequestProcessorsFactory();
};
-template
+template
void registerKeeperRequestProcessor(KeeperStorageRequestProcessorsFactory & factory)
{
- factory.registerRequest(num, [] (const Coordination::ZooKeeperRequestPtr & zk_request) { return std::make_shared(zk_request); });
+ factory.registerRequest(
+ num, [](const Coordination::ZooKeeperRequestPtr & zk_request) { return std::make_shared(zk_request); });
}
@@ -1084,13 +1547,66 @@ KeeperStorageRequestProcessorsFactory::KeeperStorageRequestProcessorsFactory()
}
-KeeperStorage::ResponsesForSessions KeeperStorage::processRequest(const Coordination::ZooKeeperRequestPtr & zk_request, int64_t session_id, int64_t time, std::optional new_last_zxid, bool check_acl)
+void KeeperStorage::preprocessRequest(
+ const Coordination::ZooKeeperRequestPtr & zk_request, int64_t session_id, int64_t time, int64_t new_last_zxid, bool check_acl)
+{
+ KeeperStorageRequestProcessorPtr request_processor = KeeperStorageRequestProcessorsFactory::instance().get(zk_request);
+
+ if (zk_request->getOpNum() == Coordination::OpNum::Close) /// Close request is special
+ {
+ auto & deltas = uncommitted_state.deltas;
+ auto session_ephemerals = ephemerals.find(session_id);
+ if (session_ephemerals != ephemerals.end())
+ {
+ for (const auto & ephemeral_path : session_ephemerals->second)
+ {
+ // For now just add deltas for removing the node
+ // On commit, ephemerals nodes will be deleted from storage
+ // and removed from the session
+ if (uncommitted_state.hasNode(ephemeral_path))
+ {
+ deltas.emplace_back(
+ parentPath(ephemeral_path).toString(),
+ new_last_zxid,
+ UpdateNodeDelta{[ephemeral_path](Node & parent)
+ {
+ --parent.stat.numChildren;
+ ++parent.stat.cversion;
+ }});
+
+ deltas.emplace_back(ephemeral_path, new_last_zxid, RemoveNodeDelta());
+ }
+ }
+ }
+
+ return;
+ }
+
+ if (check_acl && !request_processor->checkAuth(*this, session_id, false))
+ {
+ uncommitted_state.deltas.emplace_back(new_last_zxid, Coordination::Error::ZNOAUTH);
+ return;
+ }
+
+ auto new_deltas = request_processor->preprocess(*this, new_last_zxid, session_id, time);
+ uncommitted_state.deltas.insert(
+ uncommitted_state.deltas.end(), std::make_move_iterator(new_deltas.begin()), std::make_move_iterator(new_deltas.end()));
+}
+
+KeeperStorage::ResponsesForSessions KeeperStorage::processRequest(
+ const Coordination::ZooKeeperRequestPtr & zk_request,
+ int64_t session_id,
+ int64_t time,
+ std::optional new_last_zxid,
+ bool check_acl,
+ bool is_local)
{
KeeperStorage::ResponsesForSessions results;
if (new_last_zxid)
{
if (zxid >= *new_last_zxid)
- throw Exception(ErrorCodes::LOGICAL_ERROR, "Got new ZXID {} smaller or equal than current {}. It's a bug", *new_last_zxid, zxid);
+ throw Exception(
+ ErrorCodes::LOGICAL_ERROR, "Got new ZXID {} smaller or equal than current {}. It's a bug", *new_last_zxid, zxid);
zxid = *new_last_zxid;
}
@@ -1099,26 +1615,22 @@ KeeperStorage::ResponsesForSessions KeeperStorage::processRequest(const Coordina
if (zk_request->getOpNum() == Coordination::OpNum::Close) /// Close request is special
{
- auto it = ephemerals.find(session_id);
- if (it != ephemerals.end())
+ commit(zxid, session_id);
+
+ for (const auto & delta : uncommitted_state.deltas)
{
- for (const auto & ephemeral_path : it->second)
+ if (delta.zxid > zxid)
+ break;
+
+ if (std::holds_alternative(delta.operation))
{
- container.updateValue(parentPath(ephemeral_path), [&ephemeral_path] (KeeperStorage::Node & parent)
- {
- --parent.stat.numChildren;
- ++parent.stat.cversion;
- auto base_name = getBaseName(ephemeral_path);
- parent.removeChild(base_name);
- });
-
- container.erase(ephemeral_path);
-
- auto responses = processWatchesImpl(ephemeral_path, watches, list_watches, Coordination::Event::DELETED);
+ auto responses = processWatchesImpl(delta.path, watches, list_watches, Coordination::Event::DELETED);
results.insert(results.end(), responses.begin(), responses.end());
}
- ephemerals.erase(it);
}
+
+ std::erase_if(uncommitted_state.deltas, [this](const auto & delta) { return delta.zxid == zxid; });
+
clearDeadWatches(session_id);
auto auth_it = session_and_auth.find(session_id);
if (auth_it != session_and_auth.end())
@@ -1135,7 +1647,7 @@ KeeperStorage::ResponsesForSessions KeeperStorage::processRequest(const Coordina
else if (zk_request->getOpNum() == Coordination::OpNum::Heartbeat) /// Heartbeat request is also special
{
KeeperStorageRequestProcessorPtr storage_request = KeeperStorageRequestProcessorsFactory::instance().get(zk_request);
- auto [response, _] = storage_request->process(*this, zxid, session_id, time);
+ auto response = storage_request->process(*this, zxid, session_id, time);
response->xid = zk_request->xid;
response->zxid = getZXID();
@@ -1146,15 +1658,24 @@ KeeperStorage::ResponsesForSessions KeeperStorage::processRequest(const Coordina
KeeperStorageRequestProcessorPtr request_processor = KeeperStorageRequestProcessorsFactory::instance().get(zk_request);
Coordination::ZooKeeperResponsePtr response;
- if (check_acl && !request_processor->checkAuth(*this, session_id))
+ if (is_local)
{
- response = zk_request->makeResponse();
- /// Original ZooKeeper always throws no auth, even when user provided some credentials
- response->error = Coordination::Error::ZNOAUTH;
+ assert(zk_request->isReadRequest());
+ if (check_acl && !request_processor->checkAuth(*this, session_id, true))
+ {
+ response = zk_request->makeResponse();
+ /// Original ZooKeeper always throws no auth, even when user provided some credentials
+ response->error = Coordination::Error::ZNOAUTH;
+ }
+ else
+ {
+ response = request_processor->processLocal(*this, zxid, session_id, time);
+ }
}
else
{
- std::tie(response, std::ignore) = request_processor->process(*this, zxid, session_id, time);
+ response = request_processor->process(*this, zxid, session_id, time);
+ std::erase_if(uncommitted_state.deltas, [this](const auto & delta) { return delta.zxid == zxid; });
}
/// Watches for this requests are added to the watches lists
@@ -1162,7 +1683,8 @@ KeeperStorage::ResponsesForSessions KeeperStorage::processRequest(const Coordina
{
if (response->error == Coordination::Error::ZOK)
{
- auto & watches_type = zk_request->getOpNum() == Coordination::OpNum::List || zk_request->getOpNum() == Coordination::OpNum::SimpleList
+ auto & watches_type
+ = zk_request->getOpNum() == Coordination::OpNum::List || zk_request->getOpNum() == Coordination::OpNum::SimpleList
? list_watches
: watches;
@@ -1192,6 +1714,16 @@ KeeperStorage::ResponsesForSessions KeeperStorage::processRequest(const Coordina
return results;
}
+void KeeperStorage::rollbackRequest(int64_t rollback_zxid)
+{
+ // we can only rollback the last zxid (if there is any)
+ // if there is a delta with a larger zxid, we have invalid state
+ const auto last_zxid = uncommitted_state.deltas.back().zxid;
+ if (!uncommitted_state.deltas.empty() && last_zxid > rollback_zxid)
+ throw DB::Exception{DB::ErrorCodes::LOGICAL_ERROR, "Invalid state of deltas found while trying to rollback request. Last ZXID ({}) is larger than the requested ZXID ({})", last_zxid, rollback_zxid};
+
+ std::erase_if(uncommitted_state.deltas, [rollback_zxid](const auto & delta) { return delta.zxid == rollback_zxid; });
+}
void KeeperStorage::clearDeadWatches(int64_t session_id)
{
diff --git a/src/Coordination/KeeperStorage.h b/src/Coordination/KeeperStorage.h
index ccbddcf6e19..7d26ae24dd9 100644
--- a/src/Coordination/KeeperStorage.h
+++ b/src/Coordination/KeeperStorage.h
@@ -1,14 +1,14 @@
#pragma once
-#include
-#include
-#include
-#include
-#include
-#include
-#include
#include
#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
#include
@@ -29,7 +29,6 @@ struct KeeperStorageSnapshot;
class KeeperStorage
{
public:
-
struct Node
{
uint64_t acl_id = 0; /// 0 -- no ACL by default
@@ -41,26 +40,18 @@ public:
Node() : size_bytes(sizeof(Node)) { }
/// Object memory size
- uint64_t sizeInBytes() const
- {
- return size_bytes;
- }
+ uint64_t sizeInBytes() const { return size_bytes; }
void setData(String new_data);
- const auto & getData() const noexcept
- {
- return data;
- }
+ const auto & getData() const noexcept { return data; }
void addChild(StringRef child_path);
void removeChild(StringRef child_path);
- const auto & getChildren() const noexcept
- {
- return children;
- }
+ const auto & getChildren() const noexcept { return children; }
+
private:
String data;
ChildrenSet children{};
@@ -85,10 +76,7 @@ public:
std::string scheme;
std::string id;
- bool operator==(const AuthID & other) const
- {
- return scheme == other.scheme && id == other.id;
- }
+ bool operator==(const AuthID & other) const { return scheme == other.scheme && id == other.id; }
};
using RequestsForSessions = std::vector