thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-24 16:42:05 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' into add_separate_pool_for_fetches

Browse Source
This commit is contained in:
alesapin 2020-11-12 12:51:02 +03:00
commit a8d466a645
378 changed files with 4523 additions and 1811 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
.gitmodules vendored
View File

@ -190,3 +190,9 @@
path = contrib/croaring path = contrib/croaring
url = https://github.com/RoaringBitmap/CRoaring url = https://github.com/RoaringBitmap/CRoaring
branch = v0.2.66 branch = v0.2.66
[submodule "contrib/miniselect"]
path = contrib/miniselect
url = https://github.com/danlark1/miniselect
[submodule "contrib/xz"]
path = contrib/xz
url = https://github.com/xz-mirror/xz

119
CHANGELOG.md
View File

@ -1,3 +1,122 @@
### ClickHouse release v20.11.2.1, 2020-11-11
#### Backward Incompatible Change
* If some `profile` was specified in `distributed_ddl` config section, then this profile could overwrite settings of `default` profile on server startup. It's fixed, now settings of distributed DDL queries should not affect global server settings. [#16635](https://github.com/ClickHouse/ClickHouse/pull/16635) ([tavplubix](https://github.com/tavplubix)).
* Restrict to use of non-comparable data types (like `AggregateFunction`) in keys (Sorting key, Primary key, Partition key, and so on). [#16601](https://github.com/ClickHouse/ClickHouse/pull/16601) ([alesapin](https://github.com/alesapin)).
* Remove `ANALYZE` and `AST` queries, and make the setting `enable_debug_queries` obsolete since now it is the part of full featured `EXPLAIN` query. [#16536](https://github.com/ClickHouse/ClickHouse/pull/16536) ([Ivan](https://github.com/abyss7)).
* Aggregate functions `boundingRatio`, `rankCorr`, `retention`, `timeSeriesGroupSum`, `timeSeriesGroupRateSum`, `windowFunnel` were erroneously made case-insensitive. Now their names are made case sensitive as designed. Only functions that are specified in SQL standard or made for compatibility with other DBMS or functions similar to those should be case-insensitive. [#16407](https://github.com/ClickHouse/ClickHouse/pull/16407) ([alexey-milovidov](https://github.com/alexey-milovidov)).
* Make `rankCorr` function return nan on insufficient data https://github.com/ClickHouse/ClickHouse/issues/16124. [#16135](https://github.com/ClickHouse/ClickHouse/pull/16135) ([hexiaoting](https://github.com/hexiaoting)).
#### New Feature
* Added support of LDAP as a user directory for locally non-existent users. [#12736](https://github.com/ClickHouse/ClickHouse/pull/12736) ([Denis Glazachev](https://github.com/traceon)).
* Add `system.replicated_fetches` table which shows currently running background fetches. [#16428](https://github.com/ClickHouse/ClickHouse/pull/16428) ([alesapin](https://github.com/alesapin)).
* Added setting `date_time_output_format`. [#15845](https://github.com/ClickHouse/ClickHouse/pull/15845) ([Maksim Kita](https://github.com/kitaisreal)).
* Added minimal web UI to ClickHouse. [#16158](https://github.com/ClickHouse/ClickHouse/pull/16158) ([alexey-milovidov](https://github.com/alexey-milovidov)).
* Allows to read/write Single protobuf message at once (w/o length-delimiters). [#15199](https://github.com/ClickHouse/ClickHouse/pull/15199) ([filimonov](https://github.com/filimonov)).
* Added initial OpenTelemetry support. ClickHouse now accepts OpenTelemetry traceparent headers over Native and HTTP protocols, and passes them downstream in some cases. The trace spans for executed queries are saved into the `system.opentelemetry_span_log` table. [#14195](https://github.com/ClickHouse/ClickHouse/pull/14195) ([Alexander Kuzmenkov](https://github.com/akuzm)).
* Allow specify primary key in column list of `CREATE TABLE` query. This is needed for compatibility with other SQL dialects. [#15823](https://github.com/ClickHouse/ClickHouse/pull/15823) ([Maksim Kita](https://github.com/kitaisreal)).
* Implement `OFFSET offset_row_count {ROW | ROWS} FETCH {FIRST | NEXT} fetch_row_count {ROW | ROWS} {ONLY | WITH TIES}` in SELECT query with ORDER BY. This is the SQL-standard way to specify `LIMIT`. [#15855](https://github.com/ClickHouse/ClickHouse/pull/15855) ([hexiaoting](https://github.com/hexiaoting)).
* `errorCodeToName` function - return variable name of the error (useful for analyzing query_log and similar). `system.errors` table - shows how many times errors has been happened (respects `system_events_show_zero_values`). [#16438](https://github.com/ClickHouse/ClickHouse/pull/16438) ([Azat Khuzhin](https://github.com/azat)).
* Added function `untuple` which is a special function which can introduce new columns to the SELECT list by expanding a named tuple. [#16242](https://github.com/ClickHouse/ClickHouse/pull/16242) ([Nikolai Kochetov](https://github.com/KochetovNicolai), [Amos Bird](https://github.com/amosbird)).
* Now we can provide identifiers via query parameters. And these parameters can be used as table objects or columns. [#16594](https://github.com/ClickHouse/ClickHouse/pull/16594) ([Amos Bird](https://github.com/amosbird)).
* Added big integers (UInt256, Int128, Int256) and UUID data types support for MergeTree BloomFilter index. Big integers is an experimental feature. [#16642](https://github.com/ClickHouse/ClickHouse/pull/16642) ([Maksim Kita](https://github.com/kitaisreal)).
* Add `farmFingerprint64` function (non-cryptographic string hashing). [#16570](https://github.com/ClickHouse/ClickHouse/pull/16570) ([Jacob Hayes](https://github.com/JacobHayes)).
* Add `log_queries_min_query_duration_ms`, only queries slower then the value of this setting will go to `query_log`/`query_thread_log` (i.e. something like `slow_query_log` in mysql). [#16529](https://github.com/ClickHouse/ClickHouse/pull/16529) ([Azat Khuzhin](https://github.com/azat)).
* Ability to create a docker image on the top of `Alpine`. Uses precompiled binary and glibc components from ubuntu 20.04. [#16479](https://github.com/ClickHouse/ClickHouse/pull/16479) ([filimonov](https://github.com/filimonov)).
* Added `toUUIDOrNull`, `toUUIDOrZero` cast functions. [#16337](https://github.com/ClickHouse/ClickHouse/pull/16337) ([Maksim Kita](https://github.com/kitaisreal)).
* Add `max_concurrent_queries_for_all_users` setting, see [#6636](https://github.com/ClickHouse/ClickHouse/issues/6636) for use cases. [#16154](https://github.com/ClickHouse/ClickHouse/pull/16154) ([nvartolomei](https://github.com/nvartolomei)).
* Add a new option `print_query_id` to clickhouse-client. It helps generate arbitrary strings with the current query id generated by the client. Also print query id in clickhouse-client by default. [#15809](https://github.com/ClickHouse/ClickHouse/pull/15809) ([Amos Bird](https://github.com/amosbird)).
* Add `tid` and `logTrace` functions. This closes [#9434](https://github.com/ClickHouse/ClickHouse/issues/9434). [#15803](https://github.com/ClickHouse/ClickHouse/pull/15803) ([flynn](https://github.com/ucasFL)).
* Add function `formatReadableTimeDelta` that format time delta to human readable string ... [#15497](https://github.com/ClickHouse/ClickHouse/pull/15497) ([Filipe Caixeta](https://github.com/filipecaixeta)).
* Added `disable_merges` option for volumes in multi-disk configuration. [#13956](https://github.com/ClickHouse/ClickHouse/pull/13956) ([Vladimir Chebotarev](https://github.com/excitoon)).
#### Experimental Feature
* New functions `encrypt`, `aes_encrypt_mysql`, `decrypt`, `aes_decrypt_mysql`. These functions are working slowly, so we consider it as an experimental feature. [#11844](https://github.com/ClickHouse/ClickHouse/pull/11844) ([Vasily Nemkov](https://github.com/Enmk)).
#### Bug Fix
* Mask password in data_path in the `system.distribution_queue`. [#16727](https://github.com/ClickHouse/ClickHouse/pull/16727) ([Azat Khuzhin](https://github.com/azat)).
* Fix `IN` operator over several columns and tuples with enabled `transform_null_in` setting. Fixes [#15310](https://github.com/ClickHouse/ClickHouse/issues/15310). [#16722](https://github.com/ClickHouse/ClickHouse/pull/16722) ([Anton Popov](https://github.com/CurtizJ)).
* The setting `max_parallel_replicas` worked incorrectly if the queried table has no sampling. This fixes [#5733](https://github.com/ClickHouse/ClickHouse/issues/5733). [#16675](https://github.com/ClickHouse/ClickHouse/pull/16675) ([alexey-milovidov](https://github.com/alexey-milovidov)).
* Fix optimize_read_in_order/optimize_aggregation_in_order with max_threads > 0 and expression in ORDER BY. [#16637](https://github.com/ClickHouse/ClickHouse/pull/16637) ([Azat Khuzhin](https://github.com/azat)).
* Calculation of `DEFAULT` expressions was involving possible name collisions (that was very unlikely to encounter). This fixes [#9359](https://github.com/ClickHouse/ClickHouse/issues/9359). [#16612](https://github.com/ClickHouse/ClickHouse/pull/16612) ([alexey-milovidov](https://github.com/alexey-milovidov)).
* Fix `query_thread_log.query_duration_ms` unit. [#16563](https://github.com/ClickHouse/ClickHouse/pull/16563) ([Azat Khuzhin](https://github.com/azat)).
* Fix a bug when using MySQL Master -> MySQL Slave -> ClickHouse MaterializeMySQL Engine. `MaterializeMySQL` is an experimental feature. [#16504](https://github.com/ClickHouse/ClickHouse/pull/16504) ([TCeason](https://github.com/TCeason)).
* Specifically crafted argument of `round` function with `Decimal` was leading to integer division by zero. This fixes [#13338](https://github.com/ClickHouse/ClickHouse/issues/13338). [#16451](https://github.com/ClickHouse/ClickHouse/pull/16451) ([alexey-milovidov](https://github.com/alexey-milovidov)).
* Fix DROP TABLE for Distributed (racy with INSERT). [#16409](https://github.com/ClickHouse/ClickHouse/pull/16409) ([Azat Khuzhin](https://github.com/azat)).
* Fix processing of very large entries in replication queue. Very large entries may appear in ALTER queries if table structure is extremely large (near 1 MB). This fixes [#16307](https://github.com/ClickHouse/ClickHouse/issues/16307). [#16332](https://github.com/ClickHouse/ClickHouse/pull/16332) ([alexey-milovidov](https://github.com/alexey-milovidov)).
* Fixed the inconsistent behaviour when a part of return data could be dropped because the set for its filtration wasn't created. [#16308](https://github.com/ClickHouse/ClickHouse/pull/16308) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
* Fix dictGet in sharding_key (and similar places, i.e. when the function context is stored permanently). [#16205](https://github.com/ClickHouse/ClickHouse/pull/16205) ([Azat Khuzhin](https://github.com/azat)).
* Fix the exception thrown in `clickhouse-local` when trying to execute `OPTIMIZE` command. Fixes [#16076](https://github.com/ClickHouse/ClickHouse/issues/16076). [#16192](https://github.com/ClickHouse/ClickHouse/pull/16192) ([filimonov](https://github.com/filimonov)).
* Fixes [#15780](https://github.com/ClickHouse/ClickHouse/issues/15780) regression, e.g. `indexOf([1, 2, 3], toLowCardinality(1))` now is prohibited but it should not be. [#16038](https://github.com/ClickHouse/ClickHouse/pull/16038) ([Mike](https://github.com/myrrc)).
* Fix bug with MySQL database. When MySQL server used as database engine is down some queries raise Exception, because they try to get tables from disabled server, while it's unnecessary. For example, query `SELECT ... FROM system.parts` should work only with MergeTree tables and don't touch MySQL database at all. [#16032](https://github.com/ClickHouse/ClickHouse/pull/16032) ([Kruglov Pavel](https://github.com/Avogar)).
* Now exception will be thrown when `ALTER MODIFY COLUMN ... DEFAULT ...` has incompatible default with column type. Fixes [#15854](https://github.com/ClickHouse/ClickHouse/issues/15854). [#15858](https://github.com/ClickHouse/ClickHouse/pull/15858) ([alesapin](https://github.com/alesapin)).
* Fixed IPv4CIDRToRange/IPv6CIDRToRange functions to accept const IP-column values. [#15856](https://github.com/ClickHouse/ClickHouse/pull/15856) ([vladimir-golovchenko](https://github.com/vladimir-golovchenko)).
#### Improvement
* Treat `INTERVAL '1 hour'` as equivalent to `INTERVAL 1 HOUR`, to be compatible with Postgres and similar. This fixes [#15637](https://github.com/ClickHouse/ClickHouse/issues/15637). [#15978](https://github.com/ClickHouse/ClickHouse/pull/15978) ([flynn](https://github.com/ucasFL)).
* Enable parsing enum values by their numeric ids for CSV, TSV and JSON input formats. [#15685](https://github.com/ClickHouse/ClickHouse/pull/15685) ([vivarum](https://github.com/vivarum)).
* Better read task scheduling for JBOD architecture and `MergeTree` storage. New setting `read_backoff_min_concurrency` which serves as the lower limit to the number of reading threads. [#16423](https://github.com/ClickHouse/ClickHouse/pull/16423) ([Amos Bird](https://github.com/amosbird)).
* Add missing support for `LowCardinality` in `Avro` format. [#16521](https://github.com/ClickHouse/ClickHouse/pull/16521) ([Mike](https://github.com/myrrc)).
* Workaround for use `S3` with nginx server as proxy. Nginx currenty does not accept urls with empty path like `http://domain.com?delete`, but vanilla aws-sdk-cpp produces this kind of urls. This commit uses patched aws-sdk-cpp version, which makes urls with "/" as path in this cases, like `http://domain.com/?delete`. [#16814](https://github.com/ClickHouse/ClickHouse/pull/16814) ([ianton-ru](https://github.com/ianton-ru)).
* Better diagnostics on parse errors in input data. Provide row number on `Cannot read all data` errors. [#16644](https://github.com/ClickHouse/ClickHouse/pull/16644) ([alexey-milovidov](https://github.com/alexey-milovidov)).
* Make the behaviour of `minMap` and `maxMap` more desireable. It will not skip zero values in the result. Fixes [#16087](https://github.com/ClickHouse/ClickHouse/issues/16087). [#16631](https://github.com/ClickHouse/ClickHouse/pull/16631) ([Ildus Kurbangaliev](https://github.com/ildus)).
* Better update of ZooKeeper configuration in runtime. [#16630](https://github.com/ClickHouse/ClickHouse/pull/16630) ([sundyli](https://github.com/sundy-li)).
* Apply SETTINGS clause as early as possible. It allows to modify more settings in the query. This closes [#3178](https://github.com/ClickHouse/ClickHouse/issues/3178). [#16619](https://github.com/ClickHouse/ClickHouse/pull/16619) ([alexey-milovidov](https://github.com/alexey-milovidov)).
* Now `event_time_microseconds` field stores in Decimal64, not UInt64. [#16617](https://github.com/ClickHouse/ClickHouse/pull/16617) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
* Now paratmeterized functions can be used in `APPLY` column transformer. [#16589](https://github.com/ClickHouse/ClickHouse/pull/16589) ([Amos Bird](https://github.com/amosbird)).
* Improve scheduling of background task which removes data of dropped tables in `Atomic` databases. `Atomic` databases do not create broken symlink to table data directory if table actually has no data directory. [#16584](https://github.com/ClickHouse/ClickHouse/pull/16584) ([tavplubix](https://github.com/tavplubix)).
* Subqueries in `WITH` section (CTE) can reference previous subqueries in `WITH` section by their name. [#16575](https://github.com/ClickHouse/ClickHouse/pull/16575) ([Amos Bird](https://github.com/amosbird)).
* Add current_database into `system.query_thread_log`. [#16558](https://github.com/ClickHouse/ClickHouse/pull/16558) ([Azat Khuzhin](https://github.com/azat)).
* Allow to fetch parts that are already committed or outdated in the current instance into the detached directory. It's useful when migrating tables from another cluster and having N to 1 shards mapping. It's also consistent with the current fetchPartition implementation. [#16538](https://github.com/ClickHouse/ClickHouse/pull/16538) ([Amos Bird](https://github.com/amosbird)).
* Multiple improvements for `RabbitMQ`: Fixed bug for [#16263](https://github.com/ClickHouse/ClickHouse/issues/16263). Also minimized event loop lifetime. Added more efficient queues setup. [#16426](https://github.com/ClickHouse/ClickHouse/pull/16426) ([Kseniia Sumarokova](https://github.com/kssenii)).
* Fix debug assertion in `quantileDeterministic` function. In previous version it may also transfer up to two times more data over the network. Although no bug existed. This fixes [#15683](https://github.com/ClickHouse/ClickHouse/issues/15683). [#16410](https://github.com/ClickHouse/ClickHouse/pull/16410) ([alexey-milovidov](https://github.com/alexey-milovidov)).
* Add `TablesToDropQueueSize` metric. It's equal to number of dropped tables, that are waiting for background data removal. [#16364](https://github.com/ClickHouse/ClickHouse/pull/16364) ([tavplubix](https://github.com/tavplubix)).
* Better diagnostics when client has dropped connection. In previous versions, `Attempt to read after EOF` and `Broken pipe` exceptions were logged in server. In new version, it's information message `Client has dropped the connection, cancel the query.`. [#16329](https://github.com/ClickHouse/ClickHouse/pull/16329) ([alexey-milovidov](https://github.com/alexey-milovidov)).
* Add total_rows/total_bytes (from system.tables) support for Set/Join table engines. [#16306](https://github.com/ClickHouse/ClickHouse/pull/16306) ([Azat Khuzhin](https://github.com/azat)).
* Now it's possible to specify `PRIMARY KEY` without `ORDER BY` for MergeTree table engines family. Closes [#15591](https://github.com/ClickHouse/ClickHouse/issues/15591). [#16284](https://github.com/ClickHouse/ClickHouse/pull/16284) ([alesapin](https://github.com/alesapin)).
* If there is no tmp folder in the system (chroot, misconfigutation etc) `clickhouse-local` will create temporary subfolder in the current directory. [#16280](https://github.com/ClickHouse/ClickHouse/pull/16280) ([filimonov](https://github.com/filimonov)).
* Add support for nested data types (like named tuple) as sub-types. Fixes [#15587](https://github.com/ClickHouse/ClickHouse/issues/15587). [#16262](https://github.com/ClickHouse/ClickHouse/pull/16262) ([Ivan](https://github.com/abyss7)).
* Support for `database_atomic_wait_for_drop_and_detach_synchronously`/`NO DELAY`/`SYNC` for `DROP DATABASE`. [#16127](https://github.com/ClickHouse/ClickHouse/pull/16127) ([Azat Khuzhin](https://github.com/azat)).
* Add `allow_nondeterministic_optimize_skip_unused_shards` (to allow non deterministic like `rand()` or `dictGet()` in sharding key). [#16105](https://github.com/ClickHouse/ClickHouse/pull/16105) ([Azat Khuzhin](https://github.com/azat)).
* Fix `memory_profiler_step`/`max_untracked_memory` for queries via HTTP (test included). Fix the issue that adjusting this value globally in xml config does not help either, since those settings are not applied anyway, only default (4MB) value is [used](https://github.com/ClickHouse/ClickHouse/blob/17731245336d8c84f75e4c0894c5797ed7732190/src/Common/ThreadStatus.h#L104). Fix `query_id` for the most root ThreadStatus of the http query (by initializing QueryScope after reading query_id). [#16101](https://github.com/ClickHouse/ClickHouse/pull/16101) ([Azat Khuzhin](https://github.com/azat)).
* Now it's allowed to execute `ALTER ... ON CLUSTER` queries regardless of the `<internal_replication>` setting in cluster config. [#16075](https://github.com/ClickHouse/ClickHouse/pull/16075) ([alesapin](https://github.com/alesapin)).
* Fix rare issue when `clickhouse-client` may abort on exit due to loading of suggestions. This fixes [#16035](https://github.com/ClickHouse/ClickHouse/issues/16035). [#16047](https://github.com/ClickHouse/ClickHouse/pull/16047) ([alexey-milovidov](https://github.com/alexey-milovidov)).
* Add support of `cache` layout for `Redis` dictionaries with complex key. [#15985](https://github.com/ClickHouse/ClickHouse/pull/15985) ([Anton Popov](https://github.com/CurtizJ)).
* Fix query hang (endless loop) in case of misconfiguration (`connections_with_failover_max_tries` set to 0). [#15876](https://github.com/ClickHouse/ClickHouse/pull/15876) ([Azat Khuzhin](https://github.com/azat)).
* Change level of some log messages from information to debug, so information messages will not appear for every query. This closes [#5293](https://github.com/ClickHouse/ClickHouse/issues/5293). [#15816](https://github.com/ClickHouse/ClickHouse/pull/15816) ([alexey-milovidov](https://github.com/alexey-milovidov)).
* Remove `MemoryTrackingInBackground*` metrics to avoid potentially misleading results. This fixes [#15684](https://github.com/ClickHouse/ClickHouse/issues/15684). [#15813](https://github.com/ClickHouse/ClickHouse/pull/15813) ([alexey-milovidov](https://github.com/alexey-milovidov)).
* Add reconnects to `zookeeper-dump-tree` tool. [#15711](https://github.com/ClickHouse/ClickHouse/pull/15711) ([alexey-milovidov](https://github.com/alexey-milovidov)).
* Allow explicitly specify columns list in `CREATE TABLE table AS table_function(...)` query. Fixes [#9249](https://github.com/ClickHouse/ClickHouse/issues/9249) Fixes [#14214](https://github.com/ClickHouse/ClickHouse/issues/14214). [#14295](https://github.com/ClickHouse/ClickHouse/pull/14295) ([tavplubix](https://github.com/tavplubix)).
#### Performance Improvement
* Do not merge parts across partitions in SELECT FINAL. [#15938](https://github.com/ClickHouse/ClickHouse/pull/15938) ([Kruglov Pavel](https://github.com/Avogar)).
* Improve performance of `-OrNull` and `-OrDefault` aggregate functions. [#16661](https://github.com/ClickHouse/ClickHouse/pull/16661) ([alexey-milovidov](https://github.com/alexey-milovidov)).
* Improve performance of `quantileMerge`. In previous versions it was obnoxiously slow. This closes [#1463](https://github.com/ClickHouse/ClickHouse/issues/1463). [#16643](https://github.com/ClickHouse/ClickHouse/pull/16643) ([alexey-milovidov](https://github.com/alexey-milovidov)).
* Improve performance of logical functions a little. [#16347](https://github.com/ClickHouse/ClickHouse/pull/16347) ([alexey-milovidov](https://github.com/alexey-milovidov)).
* Improved performance of merges assignment in MergeTree table engines. Shouldn't be visible for the user. [#16191](https://github.com/ClickHouse/ClickHouse/pull/16191) ([alesapin](https://github.com/alesapin)).
* Speedup hashed/sparse_hashed dictionary loading by preallocating the hash table. [#15454](https://github.com/ClickHouse/ClickHouse/pull/15454) ([Azat Khuzhin](https://github.com/azat)).
* Now trivial count optimization becomes slightly non-trivial. Predicates that contain exact partition expr can be optimized too. This also fixes [#11092](https://github.com/ClickHouse/ClickHouse/issues/11092) which returns wrong count when `max_parallel_replicas > 1`. [#15074](https://github.com/ClickHouse/ClickHouse/pull/15074) ([Amos Bird](https://github.com/amosbird)).
#### Build/Testing/Packaging Improvement
* Add flaky check for stateless tests. It will detect potentially flaky functional tests in advance, before they are merged. [#16238](https://github.com/ClickHouse/ClickHouse/pull/16238) ([alesapin](https://github.com/alesapin)).
* Use proper version for `croaring` instead of amalgamation. [#16285](https://github.com/ClickHouse/ClickHouse/pull/16285) ([sundyli](https://github.com/sundy-li)).
* Improve generation of build files for `ya.make` build system (Arcadia). [#16700](https://github.com/ClickHouse/ClickHouse/pull/16700) ([alexey-milovidov](https://github.com/alexey-milovidov)).
* Add MySQL BinLog file check tool for `MaterializeMySQL` database engine. `MaterializeMySQL` is an experimental feature. [#16223](https://github.com/ClickHouse/ClickHouse/pull/16223) ([Winter Zhang](https://github.com/zhang2014)).
* Check for executable bit on non-executable files. People often accidentially commit executable files from Windows. [#15843](https://github.com/ClickHouse/ClickHouse/pull/15843) ([alexey-milovidov](https://github.com/alexey-milovidov)).
* Check for `#pragma once` in headers. [#15818](https://github.com/ClickHouse/ClickHouse/pull/15818) ([alexey-milovidov](https://github.com/alexey-milovidov)).
* Fix illegal code style `&vector[idx]` in libhdfs3. This fixes libcxx debug build. See also https://github.com/ClickHouse-Extras/libhdfs3/pull/8 . [#15815](https://github.com/ClickHouse/ClickHouse/pull/15815) ([Amos Bird](https://github.com/amosbird)).
* Fix build of one miscellaneous example tool on Mac OS. Note that we don't build examples on Mac OS in our CI (we build only ClickHouse binary), so there is zero chance it will not break again. This fixes [#15804](https://github.com/ClickHouse/ClickHouse/issues/15804). [#15808](https://github.com/ClickHouse/ClickHouse/pull/15808) ([alexey-milovidov](https://github.com/alexey-milovidov)).
* Simplify Sys/V init script. [#14135](https://github.com/ClickHouse/ClickHouse/pull/14135) ([alexey-milovidov](https://github.com/alexey-milovidov)).
* Added `boost::program_options` to `db_generator` in order to increase its usability. This closes [#15940](https://github.com/ClickHouse/ClickHouse/issues/15940). [#15973](https://github.com/ClickHouse/ClickHouse/pull/15973) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
## ClickHouse release 20.10 ## ClickHouse release 20.10
### ClickHouse release v20.10.3.30, 2020-10-28 ### ClickHouse release v20.10.3.30, 2020-10-28

1
CMakeLists.txt
View File

@ -445,6 +445,7 @@ include (cmake/find/brotli.cmake)
include (cmake/find/protobuf.cmake) include (cmake/find/protobuf.cmake)
include (cmake/find/grpc.cmake) include (cmake/find/grpc.cmake)
include (cmake/find/pdqsort.cmake) include (cmake/find/pdqsort.cmake)
include (cmake/find/miniselect.cmake)
include (cmake/find/hdfs3.cmake) # uses protobuf include (cmake/find/hdfs3.cmake) # uses protobuf
include (cmake/find/poco.cmake) include (cmake/find/poco.cmake)
include (cmake/find/curl.cmake) include (cmake/find/curl.cmake)

9
base/pcg-random/pcg_random.hpp
View File

@ -113,6 +113,12 @@
#include "pcg_extras.hpp" #include "pcg_extras.hpp"
namespace DB
{
struct PcgSerializer;
struct PcgDeserializer;
}
namespace pcg_detail { namespace pcg_detail {
using namespace pcg_extras; using namespace pcg_extras;
@ -557,6 +563,9 @@ public:
engine<xtype1, itype1, engine<xtype1, itype1,
output_mixin1, output_previous1, output_mixin1, output_previous1,
stream_mixin1, multiplier_mixin1>& rng); stream_mixin1, multiplier_mixin1>& rng);
friend ::DB::PcgSerializer;
friend ::DB::PcgDeserializer;
}; };
template <typename CharT, typename Traits, template <typename CharT, typename Traits,

2
cmake/find/miniselect.cmake Normal file
View File

@ -0,0 +1,2 @@
set(MINISELECT_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/contrib/miniselect/include)
message(STATUS "Using miniselect: ${MINISELECT_INCLUDE_DIR}")

1
contrib/CMakeLists.txt vendored
View File

@ -36,6 +36,7 @@ add_subdirectory (murmurhash)
add_subdirectory (replxx-cmake) add_subdirectory (replxx-cmake)
add_subdirectory (ryu-cmake) add_subdirectory (ryu-cmake)
add_subdirectory (unixodbc-cmake) add_subdirectory (unixodbc-cmake)
add_subdirectory (xz)
add_subdirectory (poco-cmake) add_subdirectory (poco-cmake)
add_subdirectory (croaring-cmake) add_subdirectory (croaring-cmake)

2
contrib/libunwind vendored

@ -1 +1 @@
Subproject commit 27026ef4a9c6c8cc956d1d131c4d794e24096981 Subproject commit 198458b35f100da32bd3e74c2a3ce8d236db299b

1
contrib/miniselect vendored Submodule

@ -0,0 +1 @@
Subproject commit be0af6bd0b6eb044d1acc4f754b229972d99903a

1
contrib/xz vendored Submodule

@ -0,0 +1 @@
Subproject commit 869b9d1b4edd6df07f819d360d306251f8147353

2
docker/test/fasttest/Dockerfile
View File

@ -1,5 +1,5 @@
# docker build -t yandex/clickhouse-fasttest . # docker build -t yandex/clickhouse-fasttest .
FROM ubuntu:19.10 FROM ubuntu:20.04
ENV DEBIAN_FRONTEND=noninteractive LLVM_VERSION=10 ENV DEBIAN_FRONTEND=noninteractive LLVM_VERSION=10

3
docker/test/fasttest/run.sh
View File

@ -127,7 +127,7 @@ function clone_submodules
( (
cd "$FASTTEST_SOURCE" cd "$FASTTEST_SOURCE"
SUBMODULES_TO_UPDATE=(contrib/boost contrib/zlib-ng contrib/libxml2 contrib/poco contrib/libunwind contrib/ryu contrib/fmtlib contrib/base64 contrib/cctz contrib/libcpuid contrib/double-conversion contrib/libcxx contrib/libcxxabi contrib/libc-headers contrib/lz4 contrib/zstd contrib/fastops contrib/rapidjson contrib/re2 contrib/sparsehash-c11 contrib/croaring) SUBMODULES_TO_UPDATE=(contrib/boost contrib/zlib-ng contrib/libxml2 contrib/poco contrib/libunwind contrib/ryu contrib/fmtlib contrib/base64 contrib/cctz contrib/libcpuid contrib/double-conversion contrib/libcxx contrib/libcxxabi contrib/libc-headers contrib/lz4 contrib/zstd contrib/fastops contrib/rapidjson contrib/re2 contrib/sparsehash-c11 contrib/croaring contrib/miniselect contrib/xz)
git submodule sync git submodule sync
git submodule update --init --recursive "${SUBMODULES_TO_UPDATE[@]}" git submodule update --init --recursive "${SUBMODULES_TO_UPDATE[@]}"
@ -268,6 +268,7 @@ TESTS_TO_SKIP=(
protobuf protobuf
secure secure
sha256 sha256
xz
# Not sure why these two fail even in sequential mode. Disabled for now # Not sure why these two fail even in sequential mode. Disabled for now
# to make some progress. # to make some progress.

5
docs/en/introduction/adopters.md
View File

@ -11,6 +11,7 @@ toc_title: Adopters
| Company | Industry | Usecase | Cluster Size | (Un)Compressed Data Size<abbr title="of single replica"><sup>\*</sup></abbr> | Reference | | Company | Industry | Usecase | Cluster Size | (Un)Compressed Data Size<abbr title="of single replica"><sup>\*</sup></abbr> | Reference |
|------------------------------------------------------------------------------------------------|---------------------------------|-----------------------|------------------------------------------------------------|------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |------------------------------------------------------------------------------------------------|---------------------------------|-----------------------|------------------------------------------------------------|------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| <a href="https://2gis.ru" class="favicon">2gis</a> | Maps | Monitoring | — | — | [Talk in Russian, July 2019](https://youtu.be/58sPkXfq6nw) | | <a href="https://2gis.ru" class="favicon">2gis</a> | Maps | Monitoring | — | — | [Talk in Russian, July 2019](https://youtu.be/58sPkXfq6nw) |
| <a href="https://getadmiral.com/" class="favicon">Admiral</a> | Martech | Engagement Management | — | — | [Webinar Slides, June 2020](https://altinity.com/presentations/2020/06/16/big-data-in-real-time-how-clickhouse-powers-admirals-visitor-relationships-for-publishers) |
| <a href="https://cn.aliyun.com/" class="favicon">Alibaba Cloud</a> | Cloud | Managed Service | — | — | [Official Website](https://help.aliyun.com/product/144466.html) | | <a href="https://cn.aliyun.com/" class="favicon">Alibaba Cloud</a> | Cloud | Managed Service | — | — | [Official Website](https://help.aliyun.com/product/144466.html) |
| <a href="https://alohabrowser.com/" class="favicon">Aloha Browser</a> | Mobile App | Browser backend | — | — | [Slides in Russian, May 2019](https://presentations.clickhouse.tech/meetup22/aloha.pdf) | | <a href="https://alohabrowser.com/" class="favicon">Aloha Browser</a> | Mobile App | Browser backend | — | — | [Slides in Russian, May 2019](https://presentations.clickhouse.tech/meetup22/aloha.pdf) |
| <a href="https://amadeus.com/" class="favicon">Amadeus</a> | Travel | Analytics | — | — | [Press Release, April 2018](https://www.altinity.com/blog/2018/4/5/amadeus-technologies-launches-investment-and-insights-tool-based-on-machine-learning-and-strategy-algorithms) | | <a href="https://amadeus.com/" class="favicon">Amadeus</a> | Travel | Analytics | — | — | [Press Release, April 2018](https://www.altinity.com/blog/2018/4/5/amadeus-technologies-launches-investment-and-insights-tool-based-on-machine-learning-and-strategy-algorithms) |
@ -29,6 +30,7 @@ toc_title: Adopters
| <a href="https://www.citadelsecurities.com/" class="favicon">Citadel Securities</a> | Finance | — | — | — | [Contribution, March 2019](https://github.com/ClickHouse/ClickHouse/pull/4774) | | <a href="https://www.citadelsecurities.com/" class="favicon">Citadel Securities</a> | Finance | — | — | — | [Contribution, March 2019](https://github.com/ClickHouse/ClickHouse/pull/4774) |
| <a href="https://city-mobil.ru" class="favicon">Citymobil</a> | Taxi | Analytics | — | — | [Blog Post in Russian, March 2020](https://habr.com/en/company/citymobil/blog/490660/) | | <a href="https://city-mobil.ru" class="favicon">Citymobil</a> | Taxi | Analytics | — | — | [Blog Post in Russian, March 2020](https://habr.com/en/company/citymobil/blog/490660/) |
| <a href="https://cloudflare.com" class="favicon">Cloudflare</a> | CDN | Traffic analysis | 36 servers | — | [Blog post, May 2017](https://blog.cloudflare.com/how-cloudflare-analyzes-1m-dns-queries-per-second/), [Blog post, March 2018](https://blog.cloudflare.com/http-analytics-for-6m-requests-per-second-using-clickhouse/) | | <a href="https://cloudflare.com" class="favicon">Cloudflare</a> | CDN | Traffic analysis | 36 servers | — | [Blog post, May 2017](https://blog.cloudflare.com/how-cloudflare-analyzes-1m-dns-queries-per-second/), [Blog post, March 2018](https://blog.cloudflare.com/http-analytics-for-6m-requests-per-second-using-clickhouse/) |
| <a href="https://corporate.comcast.com/" class="favicon">Comcast</a> | Media | CDN Traffic Analysis | — | — | [ApacheCon 2019 Talk](https://www.youtube.com/watch?v=e9TZ6gFDjNg) |
| <a href="https://contentsquare.com" class="favicon">ContentSquare</a> | Web analytics | Main product | — | — | [Blog post in French, November 2018](http://souslecapot.net/2018/11/21/patrick-chatain-vp-engineering-chez-contentsquare-penser-davantage-amelioration-continue-que-revolution-constante/) | | <a href="https://contentsquare.com" class="favicon">ContentSquare</a> | Web analytics | Main product | — | — | [Blog post in French, November 2018](http://souslecapot.net/2018/11/21/patrick-chatain-vp-engineering-chez-contentsquare-penser-davantage-amelioration-continue-que-revolution-constante/) |
| <a href="https://coru.net/" class="favicon">Corunet</a> | Analytics | Main product | — | — | [Slides in English, April 2019](https://github.com/ClickHouse/clickhouse-presentations/blob/master/meetup21/predictive_models.pdf) | | <a href="https://coru.net/" class="favicon">Corunet</a> | Analytics | Main product | — | — | [Slides in English, April 2019](https://github.com/ClickHouse/clickhouse-presentations/blob/master/meetup21/predictive_models.pdf) |
| <a href="https://www.creditx.com" class="favicon">CraiditX 氪信</a> | Finance AI | Analysis | — | — | [Slides in English, November 2019](https://github.com/ClickHouse/clickhouse-presentations/blob/master/meetup33/udf.pptx) | | <a href="https://www.creditx.com" class="favicon">CraiditX 氪信</a> | Finance AI | Analysis | — | — | [Slides in English, November 2019](https://github.com/ClickHouse/clickhouse-presentations/blob/master/meetup33/udf.pptx) |
@ -64,7 +66,8 @@ toc_title: Adopters
| <a href="https://tech.mymarilyn.ru" class="favicon">Marilyn</a> | Advertising | Statistics | — | — | [Talk in Russian, June 2017](https://www.youtube.com/watch?v=iXlIgx2khwc) | | <a href="https://tech.mymarilyn.ru" class="favicon">Marilyn</a> | Advertising | Statistics | — | — | [Talk in Russian, June 2017](https://www.youtube.com/watch?v=iXlIgx2khwc) |
| <a href="https://mellodesign.ru/" class="favicon">Mello</a> | Marketing | Analytics | 1 server | — | [Article, Oct 2020](https://vc.ru/marketing/166180-razrabotka-tipovogo-otcheta-skvoznoy-analitiki) | | <a href="https://mellodesign.ru/" class="favicon">Mello</a> | Marketing | Analytics | 1 server | — | [Article, Oct 2020](https://vc.ru/marketing/166180-razrabotka-tipovogo-otcheta-skvoznoy-analitiki) |
| <a href="https://www.messagebird.com" class="favicon">MessageBird</a> | Telecommunications | Statistics | — | — | [Slides in English, November 2018](https://github.com/ClickHouse/clickhouse-presentations/blob/master/meetup20/messagebird.pdf) | | <a href="https://www.messagebird.com" class="favicon">MessageBird</a> | Telecommunications | Statistics | — | — | [Slides in English, November 2018](https://github.com/ClickHouse/clickhouse-presentations/blob/master/meetup20/messagebird.pdf) |
| <a href="https://www.mindsdb.com/" class="favicon">MindsDB</a> | Machine Learning | Main Product | — | — | [Official Website](https://www.mindsdb.com/blog/machine-learning-models-as-tables-in-ch) | | <a href="https://www.mindsdb.com/" class="favicon">MindsDB</a> | Machine Learning | Main Product | — | — | [Official Website](https://www.mindsdb.com/blog/machine-learning-models-as-tables-in-ch) |x
| <a href="https://mux.com/" class="favicon">MUX</a> | Online Video | Video Analytics | — | — | [Talk in English, August 2019](https://altinity.com/presentations/2019/8/13/how-clickhouse-became-the-default-analytics-database-for-mux/) |
| <a href="https://www.mgid.com/" class="favicon">MGID</a> | Ad network | Web-analytics | — | — | [Blog post in Russian, April 2020](http://gs-studio.com/news-about-it/32777----clickhouse---c) | | <a href="https://www.mgid.com/" class="favicon">MGID</a> | Ad network | Web-analytics | — | — | [Blog post in Russian, April 2020](http://gs-studio.com/news-about-it/32777----clickhouse---c) |
| <a href="https://getnoc.com/" class="favicon">NOC Project</a> | Network Monitoring | Analytics | Main Product | — | [Official Website](https://getnoc.com/features/big-data/) | | <a href="https://getnoc.com/" class="favicon">NOC Project</a> | Network Monitoring | Analytics | Main Product | — | [Official Website](https://getnoc.com/features/big-data/) |
| <a href="https://www.nuna.com/" class="favicon">Nuna Inc.</a> | Health Data Analytics | — | — | — | [Talk in English, July 2020](https://youtu.be/GMiXCMFDMow?t=170) | | <a href="https://www.nuna.com/" class="favicon">Nuna Inc.</a> | Health Data Analytics | — | — | — | [Talk in English, July 2020](https://youtu.be/GMiXCMFDMow?t=170) |

8
docs/en/sql-reference/table-functions/null.md
View File

@ -5,7 +5,7 @@ toc_title: null function
# null {#null-function} # null {#null-function}
Accepts an inserted data of the specified structure and immediately drops it away. The function is used for convenience writing tests and demonstrations. Creates a temporary table of the specified structure with the [Null](../../engines/table-engines/special/null.md) table engine. According to the `Null`-engine properties, the table data is ignored and the table itself is immediately droped right after the query execution. The function is used for the convenience of test writing and demonstrations.
**Syntax** **Syntax**
@ -19,7 +19,7 @@ null('structure')
**Returned value** **Returned value**
A table with the specified structure, which is dropped right after the query execution. A temporary `Null`-engine table with the specified structure.
**Example** **Example**
@ -36,6 +36,8 @@ INSERT INTO t SELECT * FROM numbers_mt(1000000000);
DROP TABLE IF EXISTS t; DROP TABLE IF EXISTS t;
``` ```
See also: format **Null**. See also:
- [Null table engine](../../engines/table-engines/special/null.md)
[Original article](https://clickhouse.tech/docs/en/sql-reference/table-functions/null/) <!--hide--> [Original article](https://clickhouse.tech/docs/en/sql-reference/table-functions/null/) <!--hide-->

43
docs/ru/sql-reference/table-functions/null.md Normal file
View File

@ -0,0 +1,43 @@
---
toc_priority: 53
toc_title: null функция
---
# null {#null-function}
Создает временную таблицу указанной структуры с движком [Null](../../engines/table-engines/special/null.md). В соответствии со свойствами движка, данные в таблице игнорируются, а сама таблица удаляется сразу после выполнения запроса. Функция используется для удобства написания тестов и демонстрационных примеров.
**Синтаксис**
``` sql
null('structure')
```
**Параметр**
- `structure` — список колонок и их типов. [String](../../sql-reference/data-types/string.md).
**Возвращаемое значение**
Временная таблица указанной структуры с движком `Null`.
**Пример**
Один запрос с функцией `null`:
``` sql
INSERT INTO function null('x UInt64') SELECT * FROM numbers_mt(1000000000);
```
заменяет три запроса:
```sql
CREATE TABLE t (x UInt64) ENGINE = Null;
INSERT INTO t SELECT * FROM numbers_mt(1000000000);
DROP TABLE IF EXISTS t;
```
См. также:
- [Движок таблиц Null](../../engines/table-engines/special/null.md)
[Original article](https://clickhouse.tech/docs/en/sql-reference/table-functions/null/) <!--hide-->

2
docs/zh/engines/table-engines/log-family/index.md
View File

@ -46,6 +46,6 @@ toc_priority: 29
`Log` 引擎为表中的每一列使用不同的文件。`StripeLog` 将所有的数据存储在一个文件中。因此 `StripeLog` 引擎在操作系统中使用更少的描述符，但是 `Log` 引擎提供更高的读性能。 `Log` 引擎为表中的每一列使用不同的文件。`StripeLog` 将所有的数据存储在一个文件中。因此 `StripeLog` 引擎在操作系统中使用更少的描述符，但是 `Log` 引擎提供更高的读性能。
`TingLog` 引擎是该系列中最简单的引擎并且提供了最少的功能和最低的性能。`TingLog` 引擎不支持并行读取和并发数据访问，并将每一列存储在不同的文件中。它比其余两种支持并行读取的引擎的读取速度更慢，并且使用了和 `Log` 引擎同样多的描述符。你可以在简单的低负载的情景下使用它。 `TinyLog` 引擎是该系列中最简单的引擎并且提供了最少的功能和最低的性能。`TinyLog` 引擎不支持并行读取和并发数据访问，并将每一列存储在不同的文件中。它比其余两种支持并行读取的引擎的读取速度更慢，并且使用了和 `Log` 引擎同样多的描述符。你可以在简单的低负载的情景下使用它。
[来源文章](https://clickhouse.tech/docs/en/operations/table_engines/log_family/) <!--hide--> [来源文章](https://clickhouse.tech/docs/en/operations/table_engines/log_family/) <!--hide-->

4
docs/zh/engines/table-engines/log-family/log.md
View File

@ -1,5 +1,5 @@
# 日志 {#log} # Log {#log}
日志与 TinyLog 的不同之处在于，«标记» 的小文件与列文件存在一起。这些标记写在每个数据块上，并且包含偏移量，这些偏移量指示从哪里开始读取文件以便跳过指定的行数。这使得可以在多个线程中读取表数据。对于并发数据访问，可以同时执行读取操作，而写入操作则阻塞读取和其它写入。Log 引擎不支持索引。同样，如果写入表失败，则该表将被破坏，并且从该表读取将返回错误。Log 引擎适用于临时数据，write-once 表以及测试或演示目的。 `Log` 与 `TinyLog` 的不同之处在于，«标记» 的小文件与列文件存在一起。这些标记写在每个数据块上，并且包含偏移量，这些偏移量指示从哪里开始读取文件以便跳过指定的行数。这使得可以在多个线程中读取表数据。对于并发数据访问，可以同时执行读取操作，而写入操作则阻塞读取和其它写入。`Log`引擎不支持索引。同样，如果写入表失败，则该表将被破坏，并且从该表读取将返回错误。`Log`引擎适用于临时数据，write-once 表以及测试或演示目的。
[原始文章](https://clickhouse.tech/docs/zh/operations/table_engines/log/) <!--hide--> [原始文章](https://clickhouse.tech/docs/zh/operations/table_engines/log/) <!--hide-->

34
docs/zh/operations/tips.md
View File

@ -11,9 +11,9 @@
不要禁用超线程。 它有助于某些查询，但不适用于其他查询。 不要禁用超线程。 它有助于某些查询，但不适用于其他查询。
## 涡轮增压 {#turbo-boost} ## 超频 {#turbo-boost}
强烈推荐涡轮增压。 它显着提高了典型负载的性能。 强烈推荐超频(turbo-boost)。 它显着提高了典型负载的性能。
您可以使用 `turbostat` 要查看负载下的CPU的实际时钟速率。 您可以使用 `turbostat` 要查看负载下的CPU的实际时钟速率。
## CPU缩放调控器 {#cpu-scaling-governor} ## CPU缩放调控器 {#cpu-scaling-governor}
@ -39,18 +39,18 @@ echo 'performance' | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_gover
始终禁用交换文件。 不这样做的唯一原因是，如果您使用的ClickHouse在您的个人笔记本电脑。 始终禁用交换文件。 不这样做的唯一原因是，如果您使用的ClickHouse在您的个人笔记本电脑。
## 巨大的页面 {#huge-pages} ## 大页(Huge Pages) {#huge-pages}
始终禁用透明巨大的页面。 它会干扰内存分alloc，从而导致显着的性能下降。 始终禁用透明大页(transparent huge pages)。 它会干扰内存分alloc，从而导致显着的性能下降。
``` bash ``` bash
echo 'never' | sudo tee /sys/kernel/mm/transparent_hugepage/enabled echo 'never' | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
``` ```
使用 `perf top` 观看内核中用于内存管理的时间。 使用 `perf top` 观察内核中用于内存管理的时间。
永久巨大的页面也不需要被分配。 永久大页(permanent huge pages)也不需要被分配。
## 存储子系统 {#storage-subsystem} ## 存储系统 {#storage-subsystem}
如果您的预算允许您使用SSD，请使用SSD。 如果您的预算允许您使用SSD，请使用SSD。
如果没有，请使用硬盘。 SATA硬盘7200转就行了。 如果没有，请使用硬盘。 SATA硬盘7200转就行了。
@ -100,27 +100,27 @@ XFS也是合适的，但它还没有经过ClickHouse的彻底测试。
如果可能的话，至少使用一个10GB的网络。 1Gb也可以工作，但对于使用数十tb的数据修补副本或处理具有大量中间数据的分布式查询，情况会更糟。 如果可能的话，至少使用一个10GB的网络。 1Gb也可以工作，但对于使用数十tb的数据修补副本或处理具有大量中间数据的分布式查询，情况会更糟。
## 动物园管理员 {#zookeeper} ## Zookeeper {#zookeeper}
您可能已经将ZooKeeper用于其他目的。 您可以使用相同的zookeeper安装，如果它还没有超载。 您可能已经将ZooKeeper用于其他目的。 您可以使用相同的zookeeper安装，如果它还没有超载。
It’s best to use a fresh version of ZooKeeper – 3.4.9 or later. The version in stable Linux distributions may be outdated. 最好使用新版本的 Zookeeper – 3.4.9 或之后的版本. 稳定 Liunx 发行版中的 Zookeeper 版本可能是落后的。
You should never use manually written scripts to transfer data between different ZooKeeper clusters, because the result will be incorrect for sequential nodes. Never use the «zkcopy» utility for the same reason: https://github.com/ksprojects/zkcopy/issues/15 你永远不该使用自己手写的脚本在不同的 Zookeeper 集群之间转移数据, 这可能会导致序列节点的数据不正确。出于同样的原因，永远不要使用 zkcopy 工具: https://github.com/ksprojects/zkcopy/issues/15
如果要将现有ZooKeeper集群分为两个，正确的方法是增加其副本的数量，然后将其重新配置为两个独立的集群。 如果要将现有ZooKeeper集群分为两个，正确的方法是增加其副本的数量，然后将其重新配置为两个独立的集群。
不要在与ClickHouse相同的服务器上运行ZooKeeper。 由于ZooKeeper对延迟非常敏感，ClickHouse可能会利用所有可用的系统资源。 不要在与ClickHouse相同的服务器上运行ZooKeeper。 因为ZooKeeper对延迟非常敏感，而ClickHouse可能会占用所有可用的系统资源。
使用默认设置，ZooKeeper是一个定时炸弹: 默认设置下，ZooKeeper 就像是一个定时炸弹:
> 使用默认配置时，ZooKeeper服务器不会从旧快照和日志中删除文件（请参阅autopurge），这是操作员的责任。 当使用默认配置时，ZooKeeper服务不会从旧快照和日志中删除文件（请参阅autopurge），这是操作员的责任。
必须拆除炸弹 必须拆除炸弹。
下面的ZooKeeper（3.5.1）配置在Yandex中使用。梅地卡生产环境截至2017年5月20日: 下面的ZooKeeper（3.5.1）配置在 Yandex.Metrica 的生产环境中使用截至2017年5月20日:
动物园cfg: zoo.cfg:
``` bash ``` bash
# http://hadoop.apache.org/zookeeper/docs/current/zookeeperAdmin.html # http://hadoop.apache.org/zookeeper/docs/current/zookeeperAdmin.html
@ -222,7 +222,7 @@ JAVA_OPTS="-Xms{{ '{{' }} cluster.get('xms','128M') {{ '}}' }} \
-XX:+CMSParallelRemarkEnabled" -XX:+CMSParallelRemarkEnabled"
``` ```
盐初始化: Salt init:
description "zookeeper-{{ '{{' }} cluster['name'] {{ '}}' }} centralized coordination service" description "zookeeper-{{ '{{' }} cluster['name'] {{ '}}' }} centralized coordination service"

13
programs/client/Client.cpp
View File

@ -54,6 +54,7 @@
#include <IO/WriteHelpers.h> #include <IO/WriteHelpers.h>
#include <IO/Operators.h> #include <IO/Operators.h>
#include <IO/UseSSL.h> #include <IO/UseSSL.h>
#include <IO/WriteBufferFromOStream.h>
#include <DataStreams/AsynchronousBlockInputStream.h> #include <DataStreams/AsynchronousBlockInputStream.h>
#include <DataStreams/AddingDefaultsBlockInputStream.h> #include <DataStreams/AddingDefaultsBlockInputStream.h>
#include <DataStreams/InternalTextLogsRowOutputStream.h> #include <DataStreams/InternalTextLogsRowOutputStream.h>
@ -1158,13 +1159,13 @@ private:
ASTPtr ast_to_process; ASTPtr ast_to_process;
try try
{ {
std::stringstream dump_before_fuzz; WriteBufferFromOwnString dump_before_fuzz;
fuzz_base->dumpTree(dump_before_fuzz); fuzz_base->dumpTree(dump_before_fuzz);
auto base_before_fuzz = fuzz_base->formatForErrorMessage(); auto base_before_fuzz = fuzz_base->formatForErrorMessage();
ast_to_process = fuzz_base->clone(); ast_to_process = fuzz_base->clone();
std::stringstream dump_of_cloned_ast; WriteBufferFromOwnString dump_of_cloned_ast;
ast_to_process->dumpTree(dump_of_cloned_ast); ast_to_process->dumpTree(dump_of_cloned_ast);
// Run the original query as well. // Run the original query as well.
@ -1186,7 +1187,9 @@ private:
fprintf(stderr, "dump of cloned ast:\n%s\n", fprintf(stderr, "dump of cloned ast:\n%s\n",
dump_of_cloned_ast.str().c_str()); dump_of_cloned_ast.str().c_str());
fprintf(stderr, "dump after fuzz:\n"); fprintf(stderr, "dump after fuzz:\n");
fuzz_base->dumpTree(std::cerr); WriteBufferFromOStream cerr_buf(std::cerr, 4096);
fuzz_base->dumpTree(cerr_buf);
cerr_buf.next();
fmt::print(stderr, "IAST::clone() is broken for some AST node. This is a bug. The original AST ('dump before fuzz') and its cloned copy ('dump of cloned AST') refer to the same nodes, which must never happen. This means that their parent node doesn't implement clone() correctly."); fmt::print(stderr, "IAST::clone() is broken for some AST node. This is a bug. The original AST ('dump before fuzz') and its cloned copy ('dump of cloned AST') refer to the same nodes, which must never happen. This means that their parent node doesn't implement clone() correctly.");
@ -1529,7 +1532,9 @@ private:
if (is_interactive) if (is_interactive)
{ {
std::cout << std::endl; std::cout << std::endl;
formatAST(*res, std::cout); WriteBufferFromOStream res_buf(std::cout, 4096);
formatAST(*res, res_buf);
res_buf.next();
std::cout << std::endl << std::endl; std::cout << std::endl << std::endl;
} }

5
programs/client/QueryFuzzer.cpp
View File

@ -8,6 +8,7 @@
#include <Core/Types.h> #include <Core/Types.h>
#include <IO/Operators.h> #include <IO/Operators.h>
#include <IO/UseSSL.h> #include <IO/UseSSL.h>
#include <IO/WriteBufferFromOStream.h>
#include <Parsers/ASTExpressionList.h> #include <Parsers/ASTExpressionList.h>
#include <Parsers/ASTFunction.h> #include <Parsers/ASTFunction.h>
#include <Parsers/ASTIdentifier.h> #include <Parsers/ASTIdentifier.h>
@ -419,7 +420,9 @@ void QueryFuzzer::fuzzMain(ASTPtr & ast)
fuzz(ast); fuzz(ast);
std::cout << std::endl; std::cout << std::endl;
formatAST(*ast, std::cout, false /*highlight*/); WriteBufferFromOStream ast_buf(std::cout, 4096);
formatAST(*ast, ast_buf, false /*highlight*/);
ast_buf.next();
std::cout << std::endl << std::endl; std::cout << std::endl << std::endl;
} }

2
programs/client/Suggest.cpp
View File

@ -86,7 +86,7 @@ Suggest::Suggest()
void Suggest::loadImpl(Connection & connection, const ConnectionTimeouts & timeouts, size_t suggestion_limit) void Suggest::loadImpl(Connection & connection, const ConnectionTimeouts & timeouts, size_t suggestion_limit)
{ {
std::stringstream query; std::stringstream query; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
query << "SELECT DISTINCT arrayJoin(extractAll(name, '[\\\\w_]{2,}')) AS res FROM (" query << "SELECT DISTINCT arrayJoin(extractAll(name, '[\\\\w_]{2,}')) AS res FROM ("
"SELECT name FROM system.functions" "SELECT name FROM system.functions"
" UNION ALL " " UNION ALL "

2
programs/client/TestHint.h
View File

@ -93,7 +93,7 @@ private:
void parse(const String & hint) void parse(const String & hint)
{ {
std::stringstream ss; std::stringstream ss; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
ss << hint; ss << hint;
String item; String item;

2
programs/copier/ClusterCopier.cpp
View File

@ -162,7 +162,7 @@ void ClusterCopier::discoverShardPartitions(const ConnectionTimeouts & timeouts,
if (!missing_partitions.empty()) if (!missing_partitions.empty())
{ {
std::stringstream ss; WriteBufferFromOwnString ss;
for (const String & missing_partition : missing_partitions) for (const String & missing_partition : missing_partitions)
ss << " " << missing_partition; ss << " " << missing_partition;

2
programs/copier/Internals.cpp
View File

@ -13,7 +13,7 @@ using ConfigurationPtr = Poco::AutoPtr<Poco::Util::AbstractConfiguration>;
ConfigurationPtr getConfigurationFromXMLString(const std::string & xml_data) ConfigurationPtr getConfigurationFromXMLString(const std::string & xml_data)
{ {
std::stringstream ss(xml_data); std::stringstream ss(xml_data); // STYLE_CHECK_ALLOW_STD_STRING_STREAM
Poco::XML::InputSource input_source{ss}; Poco::XML::InputSource input_source{ss};
return {new Poco::Util::XMLConfiguration{&input_source}}; return {new Poco::Util::XMLConfiguration{&input_source}};
} }

8
programs/copier/TaskTableAndShard.h
View File

@ -394,12 +394,8 @@ inline ASTPtr TaskTable::rewriteReplicatedCreateQueryToPlain()
inline String DB::TaskShard::getDescription() const inline String DB::TaskShard::getDescription() const
{ {
std::stringstream ss; return fmt::format("N{} (having a replica {}, pull table {} of cluster {}",
ss << "N" << numberInCluster() numberInCluster(), getHostNameExample(), getQuotedTable(task_table.table_pull), task_table.cluster_pull_name);
<< " (having a replica " << getHostNameExample()
<< ", pull table " + getQuotedTable(task_table.table_pull)
<< " of cluster " + task_table.cluster_pull_name << ")";
return ss.str();
} }
inline String DB::TaskShard::getHostNameExample() const inline String DB::TaskShard::getHostNameExample() const

5
programs/format/Format.cpp
View File

@ -6,6 +6,7 @@
#include <IO/ReadBufferFromFileDescriptor.h> #include <IO/ReadBufferFromFileDescriptor.h>
#include <IO/ReadHelpers.h> #include <IO/ReadHelpers.h>
#include <IO/WriteBufferFromFileDescriptor.h> #include <IO/WriteBufferFromFileDescriptor.h>
#include <IO/WriteBufferFromOStream.h>
#include <Parsers/ParserQuery.h> #include <Parsers/ParserQuery.h>
#include <Parsers/parseQuery.h> #include <Parsers/parseQuery.h>
#include <Parsers/formatAST.h> #include <Parsers/formatAST.h>
@ -129,7 +130,9 @@ int mainEntryClickHouseFormat(int argc, char ** argv)
ASTPtr res = parseQueryAndMovePosition(parser, pos, end, "query", multiple, 0, DBMS_DEFAULT_MAX_PARSER_DEPTH); ASTPtr res = parseQueryAndMovePosition(parser, pos, end, "query", multiple, 0, DBMS_DEFAULT_MAX_PARSER_DEPTH);
if (!quiet) if (!quiet)
{ {
formatAST(*res, std::cout, hilite, oneline); WriteBufferFromOStream res_buf(std::cout, 4096);
formatAST(*res, res_buf, hilite, oneline);
res_buf.next();
if (multiple) if (multiple)
std::cout << "\n;\n"; std::cout << "\n;\n";
std::cout << std::endl; std::cout << std::endl;

2
programs/local/LocalServer.cpp
View File

@ -422,7 +422,7 @@ static const char * minimal_default_user_xml =
static ConfigurationPtr getConfigurationFromXMLString(const char * xml_data) static ConfigurationPtr getConfigurationFromXMLString(const char * xml_data)
{ {
std::stringstream ss{std::string{xml_data}}; std::stringstream ss{std::string{xml_data}}; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
Poco::XML::InputSource input_source{ss}; Poco::XML::InputSource input_source{ss};
return {new Poco::Util::XMLConfiguration{&input_source}}; return {new Poco::Util::XMLConfiguration{&input_source}};
} }

6
programs/odbc-bridge/ColumnInfoHandler.cpp
View File

@ -113,16 +113,16 @@ void ODBCColumnsInfoHandler::handleRequest(Poco::Net::HTTPServerRequest & reques
/// TODO Why not do SQLColumns instead? /// TODO Why not do SQLColumns instead?
std::string name = schema_name.empty() ? backQuoteIfNeed(table_name) : backQuoteIfNeed(schema_name) + "." + backQuoteIfNeed(table_name); std::string name = schema_name.empty() ? backQuoteIfNeed(table_name) : backQuoteIfNeed(schema_name) + "." + backQuoteIfNeed(table_name);
std::stringstream ss; WriteBufferFromOwnString buf;
std::string input = "SELECT * FROM " + name + " WHERE 1 = 0"; std::string input = "SELECT * FROM " + name + " WHERE 1 = 0";
ParserQueryWithOutput parser; ParserQueryWithOutput parser;
ASTPtr select = parseQuery(parser, input.data(), input.data() + input.size(), "", context_settings.max_query_size, context_settings.max_parser_depth); ASTPtr select = parseQuery(parser, input.data(), input.data() + input.size(), "", context_settings.max_query_size, context_settings.max_parser_depth);
IAST::FormatSettings settings(ss, true); IAST::FormatSettings settings(buf, true);
settings.always_quote_identifiers = true; settings.always_quote_identifiers = true;
settings.identifier_quoting_style = getQuotingStyle(hdbc); settings.identifier_quoting_style = getQuotingStyle(hdbc);
select->format(settings); select->format(settings);
std::string query = ss.str(); std::string query = buf.str();
LOG_TRACE(log, "Inferring structure with query '{}'", query); LOG_TRACE(log, "Inferring structure with query '{}'", query);

6
programs/odbc-bridge/ODBCBlockOutputStream.cpp
View File

@ -32,12 +32,12 @@ namespace
for (const auto & column : columns) for (const auto & column : columns)
query.columns->children.emplace_back(std::make_shared<ASTIdentifier>(column.name)); query.columns->children.emplace_back(std::make_shared<ASTIdentifier>(column.name));
std::stringstream ss; WriteBufferFromOwnString buf;
IAST::FormatSettings settings(ss, true); IAST::FormatSettings settings(buf, true);
settings.always_quote_identifiers = true; settings.always_quote_identifiers = true;
settings.identifier_quoting_style = quoting; settings.identifier_quoting_style = quoting;
query.IAST::format(settings); query.IAST::format(settings);
return ss.str(); return buf.str();
} }
std::string getQuestionMarks(size_t n) std::string getQuestionMarks(size_t n)

10
programs/server/Server.cpp
View File

@ -191,10 +191,10 @@ int Server::run()
if (config().hasOption("help")) if (config().hasOption("help"))
{ {
Poco::Util::HelpFormatter help_formatter(Server::options()); Poco::Util::HelpFormatter help_formatter(Server::options());
std::stringstream header; auto header_str = fmt::format("{} [OPTION] [-- [ARG]...]\n"
header << commandName() << " [OPTION] [-- [ARG]...]\n"; "positional arguments can be used to rewrite config.xml properties, for example, --http_port=8010",
header << "positional arguments can be used to rewrite config.xml properties, for example, --http_port=8010"; commandName());
help_formatter.setHeader(header.str()); help_formatter.setHeader(header_str);
help_formatter.format(std::cout); help_formatter.format(std::cout);
return 0; return 0;
} }
@ -568,6 +568,8 @@ int Server::main(const std::vector<std::string> & /*args*/)
if (config->has("zookeeper")) if (config->has("zookeeper"))
global_context->reloadZooKeeperIfChanged(config); global_context->reloadZooKeeperIfChanged(config);
global_context->reloadAuxiliaryZooKeepersConfigIfChanged(config);
global_context->updateStorageConfiguration(*config); global_context->updateStorageConfiguration(*config);
}, },
/* already_loaded = */ true); /* already_loaded = */ true);

12
src/Access/DiskAccessStorage.cpp
View File

@ -197,11 +197,13 @@ namespace
boost::range::push_back(queries, InterpreterShowGrantsQuery::getAttachGrantQueries(entity)); boost::range::push_back(queries, InterpreterShowGrantsQuery::getAttachGrantQueries(entity));
/// Serialize the list of ATTACH queries to a string. /// Serialize the list of ATTACH queries to a string.
std::stringstream ss; WriteBufferFromOwnString buf;
ss.exceptions(std::ios::failbit);
for (const ASTPtr & query : queries) for (const ASTPtr & query : queries)
ss << *query << ";\n"; {
String file_contents = std::move(ss).str(); formatAST(*query, buf, false, true);
buf.write(";\n", 2);
}
String file_contents = buf.str();
/// First we save *.tmp file and then we rename if everything's ok. /// First we save *.tmp file and then we rename if everything's ok.
auto tmp_file_path = std::filesystem::path{file_path}.replace_extension(".tmp"); auto tmp_file_path = std::filesystem::path{file_path}.replace_extension(".tmp");
@ -353,7 +355,7 @@ String DiskAccessStorage::getStorageParamsJSON() const
json.set("path", directory_path); json.set("path", directory_path);
if (readonly) if (readonly)
json.set("readonly", readonly.load()); json.set("readonly", readonly.load());
std::ostringstream oss; std::ostringstream oss; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
oss.exceptions(std::ios::failbit); oss.exceptions(std::ios::failbit);
Poco::JSON::Stringifier::stringify(json, oss); Poco::JSON::Stringifier::stringify(json, oss);
return oss.str(); return oss.str();

2
src/Access/LDAPAccessStorage.cpp
View File

@ -150,7 +150,7 @@ String LDAPAccessStorage::getStorageParamsJSON() const
params_json.set("server", ldap_server); params_json.set("server", ldap_server);
params_json.set("roles", default_role_names); params_json.set("roles", default_role_names);
std::ostringstream oss; std::ostringstream oss; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
oss.exceptions(std::ios::failbit); oss.exceptions(std::ios::failbit);
Poco::JSON::Stringifier::stringify(params_json, oss); Poco::JSON::Stringifier::stringify(params_json, oss);

2
src/Access/UsersConfigAccessStorage.cpp
View File

@ -460,7 +460,7 @@ String UsersConfigAccessStorage::getStorageParamsJSON() const
Poco::JSON::Object json; Poco::JSON::Object json;
if (!path.empty()) if (!path.empty())
json.set("path", path); json.set("path", path);
std::ostringstream oss; std::ostringstream oss; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
oss.exceptions(std::ios::failbit); oss.exceptions(std::ios::failbit);
Poco::JSON::Stringifier::stringify(json, oss); Poco::JSON::Stringifier::stringify(json, oss);
return oss.str(); return oss.str();

27
src/AggregateFunctions/AggregateFunctionGroupArray.h
View File

@ -2,6 +2,9 @@
#include <IO/ReadHelpers.h> #include <IO/ReadHelpers.h>
#include <IO/WriteHelpers.h> #include <IO/WriteHelpers.h>
#include <IO/ReadBufferFromString.h>
#include <IO/WriteBufferFromString.h>
#include <IO/Operators.h>
#include <DataTypes/DataTypeArray.h> #include <DataTypes/DataTypeArray.h>
#include <DataTypes/DataTypeString.h> #include <DataTypes/DataTypeString.h>
@ -244,10 +247,9 @@ public:
if constexpr (Trait::sampler == Sampler::RNG) if constexpr (Trait::sampler == Sampler::RNG)
{ {
DB::writeIntBinary<size_t>(this->data(place).total_values, buf); DB::writeIntBinary<size_t>(this->data(place).total_values, buf);
std::ostringstream rng_stream; WriteBufferFromOwnString rng_buf;
rng_stream.exceptions(std::ios::failbit); rng_buf << this->data(place).rng;
rng_stream << this->data(place).rng; DB::writeStringBinary(rng_buf.str(), buf);
DB::writeStringBinary(rng_stream.str(), buf);
} }
// TODO // TODO
@ -275,9 +277,8 @@ public:
DB::readIntBinary<size_t>(this->data(place).total_values, buf); DB::readIntBinary<size_t>(this->data(place).total_values, buf);
std::string rng_string; std::string rng_string;
DB::readStringBinary(rng_string, buf); DB::readStringBinary(rng_string, buf);
std::istringstream rng_stream(rng_string); ReadBufferFromString rng_buf(rng_string);
rng_stream.exceptions(std::ios::failbit); rng_buf >> this->data(place).rng;
rng_stream >> this->data(place).rng;
} }
// TODO // TODO
@ -565,10 +566,9 @@ public:
if constexpr (Trait::sampler == Sampler::RNG) if constexpr (Trait::sampler == Sampler::RNG)
{ {
DB::writeIntBinary<size_t>(data(place).total_values, buf); DB::writeIntBinary<size_t>(data(place).total_values, buf);
std::ostringstream rng_stream; WriteBufferFromOwnString rng_buf;
rng_stream.exceptions(std::ios::failbit); rng_buf << data(place).rng;
rng_stream << data(place).rng; DB::writeStringBinary(rng_buf.str(), buf);
DB::writeStringBinary(rng_stream.str(), buf);
} }
// TODO // TODO
@ -600,9 +600,8 @@ public:
DB::readIntBinary<size_t>(data(place).total_values, buf); DB::readIntBinary<size_t>(data(place).total_values, buf);
std::string rng_string; std::string rng_string;
DB::readStringBinary(rng_string, buf); DB::readStringBinary(rng_string, buf);
std::istringstream rng_stream(rng_string); ReadBufferFromString rng_buf(rng_string);
rng_stream.exceptions(std::ios::failbit); rng_buf >> data(place).rng;
rng_stream >> data(place).rng;
} }
// TODO // TODO

2
src/AggregateFunctions/AggregateFunctionRetention.h
View File

@ -1,7 +1,5 @@
#pragma once #pragma once
#include <iostream>
#include <sstream>
#include <unordered_set> #include <unordered_set>
#include <Columns/ColumnsNumber.h> #include <Columns/ColumnsNumber.h>
#include <Columns/ColumnArray.h> #include <Columns/ColumnArray.h>

2
src/AggregateFunctions/AggregateFunctionTimeSeriesGroupSum.h
View File

@ -1,10 +1,8 @@
#pragma once #pragma once
#include <bitset> #include <bitset>
#include <iostream>
#include <map> #include <map>
#include <queue> #include <queue>
#include <sstream>
#include <unordered_set> #include <unordered_set>
#include <utility> #include <utility>
#include <Columns/ColumnArray.h> #include <Columns/ColumnArray.h>

2
src/AggregateFunctions/AggregateFunctionWindowFunnel.h
View File

@ -1,7 +1,5 @@
#pragma once #pragma once
#include <iostream>
#include <sstream>
#include <unordered_set> #include <unordered_set>
#include <Columns/ColumnsNumber.h> #include <Columns/ColumnsNumber.h>
#include <DataTypes/DataTypeDateTime.h> #include <DataTypes/DataTypeDateTime.h>

29
src/AggregateFunctions/QuantileExact.h
View File

@ -8,6 +8,9 @@
#include <Common/NaNUtils.h> #include <Common/NaNUtils.h>
#include <Common/PODArray.h> #include <Common/PODArray.h>
#if !defined(ARCADIA_BUILD)
#include <miniselect/floyd_rivest_select.h> // Y_IGNORE
#endif
namespace DB namespace DB
{ {
@ -87,7 +90,11 @@ struct QuantileExact : QuantileExactBase<Value, QuantileExact<Value>>
{ {
size_t n = level < 1 ? level * array.size() : (array.size() - 1); size_t n = level < 1 ? level * array.size() : (array.size() - 1);
#if !defined(ARCADIA_BUILD)
miniselect::floyd_rivest_select(array.begin(), array.begin() + n, array.end()); /// NOTE You can think of the radix-select algorithm.
#else
std::nth_element(array.begin(), array.begin() + n, array.end()); /// NOTE You can think of the radix-select algorithm. std::nth_element(array.begin(), array.begin() + n, array.end()); /// NOTE You can think of the radix-select algorithm.
#endif
return array[n]; return array[n];
} }
@ -107,8 +114,11 @@ struct QuantileExact : QuantileExactBase<Value, QuantileExact<Value>>
size_t n = level < 1 ? level * array.size() : (array.size() - 1); size_t n = level < 1 ? level * array.size() : (array.size() - 1);
#if !defined(ARCADIA_BUILD)
miniselect::floyd_rivest_select(array.begin() + prev_n, array.begin() + n, array.end());
#else
std::nth_element(array.begin() + prev_n, array.begin() + n, array.end()); std::nth_element(array.begin() + prev_n, array.begin() + n, array.end());
#endif
result[indices[i]] = array[n]; result[indices[i]] = array[n];
prev_n = n; prev_n = n;
} }
@ -144,7 +154,11 @@ struct QuantileExactExclusive : public QuantileExact<Value>
else if (n < 1) else if (n < 1)
return static_cast<Float64>(array[0]); return static_cast<Float64>(array[0]);
#if !defined(ARCADIA_BUILD)
miniselect::floyd_rivest_select(array.begin(), array.begin() + n - 1, array.end());
#else
std::nth_element(array.begin(), array.begin() + n - 1, array.end()); std::nth_element(array.begin(), array.begin() + n - 1, array.end());
#endif
auto nth_element = std::min_element(array.begin() + n, array.end()); auto nth_element = std::min_element(array.begin() + n, array.end());
return static_cast<Float64>(array[n - 1]) + (h - n) * static_cast<Float64>(*nth_element - array[n - 1]); return static_cast<Float64>(array[n - 1]) + (h - n) * static_cast<Float64>(*nth_element - array[n - 1]);
@ -173,7 +187,11 @@ struct QuantileExactExclusive : public QuantileExact<Value>
result[indices[i]] = static_cast<Float64>(array[0]); result[indices[i]] = static_cast<Float64>(array[0]);
else else
{ {
#if !defined(ARCADIA_BUILD)
miniselect::floyd_rivest_select(array.begin() + prev_n, array.begin() + n - 1, array.end());
#else
std::nth_element(array.begin() + prev_n, array.begin() + n - 1, array.end()); std::nth_element(array.begin() + prev_n, array.begin() + n - 1, array.end());
#endif
auto nth_element = std::min_element(array.begin() + n, array.end()); auto nth_element = std::min_element(array.begin() + n, array.end());
result[indices[i]] = static_cast<Float64>(array[n - 1]) + (h - n) * static_cast<Float64>(*nth_element - array[n - 1]); result[indices[i]] = static_cast<Float64>(array[n - 1]) + (h - n) * static_cast<Float64>(*nth_element - array[n - 1]);
@ -208,8 +226,11 @@ struct QuantileExactInclusive : public QuantileExact<Value>
return static_cast<Float64>(array[array.size() - 1]); return static_cast<Float64>(array[array.size() - 1]);
else if (n < 1) else if (n < 1)
return static_cast<Float64>(array[0]); return static_cast<Float64>(array[0]);
#if !defined(ARCADIA_BUILD)
miniselect::floyd_rivest_select(array.begin(), array.begin() + n - 1, array.end());
#else
std::nth_element(array.begin(), array.begin() + n - 1, array.end()); std::nth_element(array.begin(), array.begin() + n - 1, array.end());
#endif
auto nth_element = std::min_element(array.begin() + n, array.end()); auto nth_element = std::min_element(array.begin() + n, array.end());
return static_cast<Float64>(array[n - 1]) + (h - n) * static_cast<Float64>(*nth_element - array[n - 1]); return static_cast<Float64>(array[n - 1]) + (h - n) * static_cast<Float64>(*nth_element - array[n - 1]);
@ -236,7 +257,11 @@ struct QuantileExactInclusive : public QuantileExact<Value>
result[indices[i]] = static_cast<Float64>(array[0]); result[indices[i]] = static_cast<Float64>(array[0]);
else else
{ {
#if !defined(ARCADIA_BUILD)
miniselect::floyd_rivest_select(array.begin() + prev_n, array.begin() + n - 1, array.end());
#else
std::nth_element(array.begin() + prev_n, array.begin() + n - 1, array.end()); std::nth_element(array.begin() + prev_n, array.begin() + n - 1, array.end());
#endif
auto nth_element = std::min_element(array.begin() + n, array.end()); auto nth_element = std::min_element(array.begin() + n, array.end());
result[indices[i]] = static_cast<Float64>(array[n - 1]) + (h - n) * static_cast<Float64>(*nth_element - array[n - 1]); result[indices[i]] = static_cast<Float64>(array[n - 1]) + (h - n) * static_cast<Float64>(*nth_element - array[n - 1]);

11
src/AggregateFunctions/QuantileTiming.h
View File

@ -7,6 +7,9 @@
#include <IO/ReadHelpers.h> #include <IO/ReadHelpers.h>
#include <IO/WriteHelpers.h> #include <IO/WriteHelpers.h>
#if !defined(ARCADIA_BUILD)
#include <miniselect/floyd_rivest_select.h> // Y_IGNORE
#endif
namespace DB namespace DB
{ {
@ -179,7 +182,11 @@ namespace detail
/// Sorting an array will not be considered a violation of constancy. /// Sorting an array will not be considered a violation of constancy.
auto & array = elems; auto & array = elems;
#if !defined(ARCADIA_BUILD)
miniselect::floyd_rivest_select(array.begin(), array.begin() + n, array.end());
#else
std::nth_element(array.begin(), array.begin() + n, array.end()); std::nth_element(array.begin(), array.begin() + n, array.end());
#endif
quantile = array[n]; quantile = array[n];
} }
@ -200,7 +207,11 @@ namespace detail
? level * elems.size() ? level * elems.size()
: (elems.size() - 1); : (elems.size() - 1);
#if !defined(ARCADIA_BUILD)
miniselect::floyd_rivest_select(array.begin() + prev_n, array.begin() + n, array.end());
#else
std::nth_element(array.begin() + prev_n, array.begin() + n, array.end()); std::nth_element(array.begin() + prev_n, array.begin() + n, array.end());
#endif
result[level_index] = array[n]; result[level_index] = array[n];
prev_n = n; prev_n = n;

16
src/AggregateFunctions/ReservoirSampler.h
View File

@ -3,11 +3,13 @@
#include <limits> #include <limits>
#include <algorithm> #include <algorithm>
#include <climits> #include <climits>
#include <sstream>
#include <common/types.h> #include <common/types.h>
#include <IO/ReadBuffer.h> #include <IO/ReadBuffer.h>
#include <IO/ReadHelpers.h> #include <IO/ReadHelpers.h>
#include <IO/WriteHelpers.h> #include <IO/WriteHelpers.h>
#include <IO/ReadBufferFromString.h>
#include <IO/WriteBufferFromString.h>
#include <IO/Operators.h>
#include <Common/PODArray.h> #include <Common/PODArray.h>
#include <Common/NaNUtils.h> #include <Common/NaNUtils.h>
#include <Poco/Exception.h> #include <Poco/Exception.h>
@ -190,9 +192,8 @@ public:
std::string rng_string; std::string rng_string;
DB::readStringBinary(rng_string, buf); DB::readStringBinary(rng_string, buf);
std::istringstream rng_stream(rng_string); DB::ReadBufferFromString rng_buf(rng_string);
rng_stream.exceptions(std::ios::failbit); rng_buf >> rng;
rng_stream >> rng;
for (size_t i = 0; i < samples.size(); ++i) for (size_t i = 0; i < samples.size(); ++i)
DB::readBinary(samples[i], buf); DB::readBinary(samples[i], buf);
@ -205,10 +206,9 @@ public:
DB::writeIntBinary<size_t>(sample_count, buf); DB::writeIntBinary<size_t>(sample_count, buf);
DB::writeIntBinary<size_t>(total_values, buf); DB::writeIntBinary<size_t>(total_values, buf);
std::ostringstream rng_stream; DB::WriteBufferFromOwnString rng_buf;
rng_stream.exceptions(std::ios::failbit); rng_buf << rng;
rng_stream << rng; DB::writeStringBinary(rng_buf.str(), buf);
DB::writeStringBinary(rng_stream.str(), buf);
for (size_t i = 0; i < std::min(sample_count, total_values); ++i) for (size_t i = 0; i < std::min(sample_count, total_values); ++i)
DB::writeBinary(samples[i], buf); DB::writeBinary(samples[i], buf);

1
src/AggregateFunctions/ReservoirSamplerDeterministic.h
View File

@ -3,7 +3,6 @@
#include <limits> #include <limits>
#include <algorithm> #include <algorithm>
#include <climits> #include <climits>
#include <sstream>
#include <AggregateFunctions/ReservoirSampler.h> #include <AggregateFunctions/ReservoirSampler.h>
#include <common/types.h> #include <common/types.h>
#include <Common/HashTable/Hash.h> #include <Common/HashTable/Hash.h>

8
src/CMakeLists.txt
View File

@ -321,6 +321,7 @@ target_include_directories(clickhouse_common_io PUBLIC ${CMAKE_CURRENT_BINARY_DI
dbms_target_include_directories(PUBLIC ${CMAKE_CURRENT_BINARY_DIR}/Core/include) dbms_target_include_directories(PUBLIC ${CMAKE_CURRENT_BINARY_DIR}/Core/include)
dbms_target_include_directories(SYSTEM BEFORE PUBLIC ${PDQSORT_INCLUDE_DIR}) dbms_target_include_directories(SYSTEM BEFORE PUBLIC ${PDQSORT_INCLUDE_DIR})
dbms_target_include_directories(SYSTEM BEFORE PUBLIC ${MINISELECT_INCLUDE_DIR})
if (ZSTD_LIBRARY) if (ZSTD_LIBRARY)
dbms_target_link_libraries(PRIVATE ${ZSTD_LIBRARY}) dbms_target_link_libraries(PRIVATE ${ZSTD_LIBRARY})
@ -329,6 +330,13 @@ if (ZSTD_LIBRARY)
endif () endif ()
endif() endif()
set (LZMA_LIBRARY liblzma)
set (LZMA_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/contrib/xz/src/liblzma/api)
if (LZMA_LIBRARY)
target_link_libraries (clickhouse_common_io PUBLIC ${LZMA_LIBRARY})
target_include_directories (clickhouse_common_io SYSTEM BEFORE PUBLIC ${LZMA_INCLUDE_DIR})
endif()
if (USE_ICU) if (USE_ICU)
dbms_target_link_libraries (PRIVATE ${ICU_LIBRARIES}) dbms_target_link_libraries (PRIVATE ${ICU_LIBRARIES})
dbms_target_include_directories (SYSTEM PRIVATE ${ICU_INCLUDE_DIRS}) dbms_target_include_directories (SYSTEM PRIVATE ${ICU_INCLUDE_DIRS})

8
src/Client/MultiplexedConnections.cpp
View File

@ -1,5 +1,6 @@
#include <Client/MultiplexedConnections.h> #include <Client/MultiplexedConnections.h>
#include <IO/ConnectionTimeouts.h> #include <IO/ConnectionTimeouts.h>
#include <IO/Operators.h>
#include <Common/thread_local_rng.h> #include <Common/thread_local_rng.h>
@ -222,19 +223,18 @@ std::string MultiplexedConnections::dumpAddresses() const
std::string MultiplexedConnections::dumpAddressesUnlocked() const std::string MultiplexedConnections::dumpAddressesUnlocked() const
{ {
bool is_first = true; bool is_first = true;
std::ostringstream os; WriteBufferFromOwnString buf;
os.exceptions(std::ios::failbit);
for (const ReplicaState & state : replica_states) for (const ReplicaState & state : replica_states)
{ {
const Connection * connection = state.connection; const Connection * connection = state.connection;
if (connection) if (connection)
{ {
os << (is_first ? "" : "; ") << connection->getDescription(); buf << (is_first ? "" : "; ") << connection->getDescription();
is_first = false; is_first = false;
} }
} }
return os.str(); return buf.str();
} }
Packet MultiplexedConnections::receivePacketUnlocked() Packet MultiplexedConnections::receivePacketUnlocked()

13
src/Columns/ColumnArray.cpp
View File

@ -20,6 +20,10 @@
#include <Common/WeakHash.h> #include <Common/WeakHash.h>
#include <Common/HashTable/Hash.h> #include <Common/HashTable/Hash.h>
#if !defined(ARCADIA_BUILD)
#include <miniselect/floyd_rivest_select.h> // Y_IGNORE
#endif
namespace DB namespace DB
{ {
@ -782,7 +786,11 @@ void ColumnArray::getPermutationImpl(size_t limit, Permutation & res, Comparator
auto less = [&cmp](size_t lhs, size_t rhs){ return cmp(lhs, rhs) < 0; }; auto less = [&cmp](size_t lhs, size_t rhs){ return cmp(lhs, rhs) < 0; };
if (limit) if (limit)
#if !defined(ARCADIA_BUILD)
miniselect::floyd_rivest_partial_sort(res.begin(), res.begin() + limit, res.end(), less);
#else
std::partial_sort(res.begin(), res.begin() + limit, res.end(), less); std::partial_sort(res.begin(), res.begin() + limit, res.end(), less);
#endif
else else
std::sort(res.begin(), res.end(), less); std::sort(res.begin(), res.end(), less);
} }
@ -834,8 +842,11 @@ void ColumnArray::updatePermutationImpl(size_t limit, Permutation & res, EqualRa
return; return;
/// Since then we are working inside the interval. /// Since then we are working inside the interval.
#if !defined(ARCADIA_BUILD)
miniselect::floyd_rivest_partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, less);
#else
std::partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, less); std::partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, less);
#endif
auto new_first = first; auto new_first = first;
for (auto j = first + 1; j < limit; ++j) for (auto j = first + 1; j < limit; ++j)
{ {

21
src/Columns/ColumnDecimal.cpp
View File

@ -8,6 +8,10 @@
#include <common/unaligned.h> #include <common/unaligned.h>
#include <ext/scope_guard.h> #include <ext/scope_guard.h>
#if !defined(ARCADIA_BUILD)
#include <miniselect/floyd_rivest_select.h> // Y_IGNORE
#endif
#include <IO/WriteHelpers.h> #include <IO/WriteHelpers.h>
@ -162,10 +166,10 @@ void ColumnDecimal<T>::updatePermutation(bool reverse, size_t limit, int, IColum
{ {
const auto& [first, last] = equal_ranges[i]; const auto& [first, last] = equal_ranges[i];
if (reverse) if (reverse)
std::partial_sort(res.begin() + first, res.begin() + last, res.begin() + last, std::sort(res.begin() + first, res.begin() + last,
[this](size_t a, size_t b) { return data[a] > data[b]; }); [this](size_t a, size_t b) { return data[a] > data[b]; });
else else
std::partial_sort(res.begin() + first, res.begin() + last, res.begin() + last, std::sort(res.begin() + first, res.begin() + last,
[this](size_t a, size_t b) { return data[a] < data[b]; }); [this](size_t a, size_t b) { return data[a] < data[b]; });
auto new_first = first; auto new_first = first;
@ -193,12 +197,21 @@ void ColumnDecimal<T>::updatePermutation(bool reverse, size_t limit, int, IColum
/// Since then we are working inside the interval. /// Since then we are working inside the interval.
if (reverse) if (reverse)
#if !defined(ARCADIA_BUILD)
miniselect::floyd_rivest_partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last,
[this](size_t a, size_t b) { return data[a] > data[b]; });
#else
std::partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, std::partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last,
[this](size_t a, size_t b) { return data[a] > data[b]; }); [this](size_t a, size_t b) { return data[a] > data[b]; });
#endif
else else
std::partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, #if !defined(ARCADIA_BUILD)
miniselect::floyd_rivest_partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last,
[this](size_t a, size_t b) { return data[a] < data[b]; }); [this](size_t a, size_t b) { return data[a] < data[b]; });
#else
std::partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last,
[this](size_t a, size_t b) { return data[a] > data[b]; });
#endif
auto new_first = first; auto new_first = first;
for (auto j = first + 1; j < limit; ++j) for (auto j = first + 1; j < limit; ++j)
{ {

11
src/Columns/ColumnDecimal.h
View File

@ -7,6 +7,9 @@
#include <Columns/IColumnImpl.h> #include <Columns/IColumnImpl.h>
#include <Columns/ColumnVectorHelper.h> #include <Columns/ColumnVectorHelper.h>
#include <Core/Field.h> #include <Core/Field.h>
#if !defined(ARCADIA_BUILD)
#include <miniselect/floyd_rivest_select.h> // Y_IGNORE
#endif
namespace DB namespace DB
@ -253,9 +256,17 @@ protected:
sort_end = res.begin() + limit; sort_end = res.begin() + limit;
if (reverse) if (reverse)
#if !defined(ARCADIA_BUILD)
miniselect::floyd_rivest_partial_sort(res.begin(), sort_end, res.end(), [this](size_t a, size_t b) { return data[a] > data[b]; });
#else
std::partial_sort(res.begin(), sort_end, res.end(), [this](size_t a, size_t b) { return data[a] > data[b]; }); std::partial_sort(res.begin(), sort_end, res.end(), [this](size_t a, size_t b) { return data[a] > data[b]; });
#endif
else else
#if !defined(ARCADIA_BUILD)
miniselect::floyd_rivest_partial_sort(res.begin(), sort_end, res.end(), [this](size_t a, size_t b) { return data[a] < data[b]; });
#else
std::partial_sort(res.begin(), sort_end, res.end(), [this](size_t a, size_t b) { return data[a] < data[b]; }); std::partial_sort(res.begin(), sort_end, res.end(), [this](size_t a, size_t b) { return data[a] < data[b]; });
#endif
} }
}; };

19
src/Columns/ColumnFixedString.cpp
View File

@ -10,6 +10,9 @@
#include <Common/HashTable/Hash.h> #include <Common/HashTable/Hash.h>
#include <ext/scope_guard.h> #include <ext/scope_guard.h>
#if !defined(ARCADIA_BUILD)
#include <miniselect/floyd_rivest_select.h> // Y_IGNORE
#endif
#include <DataStreams/ColumnGathererStream.h> #include <DataStreams/ColumnGathererStream.h>
@ -157,9 +160,17 @@ void ColumnFixedString::getPermutation(bool reverse, size_t limit, int /*nan_dir
if (limit) if (limit)
{ {
if (reverse) if (reverse)
#if !defined(ARCADIA_BUILD)
miniselect::floyd_rivest_partial_sort(res.begin(), res.begin() + limit, res.end(), less<false>(*this));
#else
std::partial_sort(res.begin(), res.begin() + limit, res.end(), less<false>(*this)); std::partial_sort(res.begin(), res.begin() + limit, res.end(), less<false>(*this));
#endif
else else
#if !defined(ARCADIA_BUILD)
miniselect::floyd_rivest_partial_sort(res.begin(), res.begin() + limit, res.end(), less<true>(*this));
#else
std::partial_sort(res.begin(), res.begin() + limit, res.end(), less<true>(*this)); std::partial_sort(res.begin(), res.begin() + limit, res.end(), less<true>(*this));
#endif
} }
else else
{ {
@ -217,9 +228,17 @@ void ColumnFixedString::updatePermutation(bool reverse, size_t limit, int, Permu
/// Since then we are working inside the interval. /// Since then we are working inside the interval.
if (reverse) if (reverse)
#if !defined(ARCADIA_BUILD)
miniselect::floyd_rivest_partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, less<false>(*this));
#else
std::partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, less<false>(*this)); std::partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, less<false>(*this));
#endif
else else
#if !defined(ARCADIA_BUILD)
miniselect::floyd_rivest_partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, less<true>(*this));
#else
std::partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, less<true>(*this)); std::partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, less<true>(*this));
#endif
auto new_first = first; auto new_first = first;
for (auto j = first + 1; j < limit; ++j) for (auto j = first + 1; j < limit; ++j)

8
src/Columns/ColumnLowCardinality.cpp
View File

@ -9,6 +9,10 @@
#include <ext/scope_guard.h> #include <ext/scope_guard.h>
#if !defined(ARCADIA_BUILD)
#include <miniselect/floyd_rivest_select.h> // Y_IGNORE
#endif
namespace DB namespace DB
{ {
namespace ErrorCodes namespace ErrorCodes
@ -393,7 +397,11 @@ void ColumnLowCardinality::updatePermutationImpl(size_t limit, Permutation & res
/// Since then we are working inside the interval. /// Since then we are working inside the interval.
#if !defined(ARCADIA_BUILD)
miniselect::floyd_rivest_partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, less);
#else
std::partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, less); std::partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, less);
#endif
auto new_first = first; auto new_first = first;
for (auto j = first + 1; j < limit; ++j) for (auto j = first + 1; j < limit; ++j)

13
src/Columns/ColumnString.cpp
View File

@ -10,6 +10,10 @@
#include <common/unaligned.h> #include <common/unaligned.h>
#include <ext/scope_guard.h> #include <ext/scope_guard.h>
#if !defined(ARCADIA_BUILD)
#include <miniselect/floyd_rivest_select.h> // Y_IGNORE
#endif
namespace DB namespace DB
{ {
@ -313,7 +317,11 @@ void ColumnString::getPermutationImpl(size_t limit, Permutation & res, Comparato
auto less = [&cmp](size_t lhs, size_t rhs){ return cmp(lhs, rhs) < 0; }; auto less = [&cmp](size_t lhs, size_t rhs){ return cmp(lhs, rhs) < 0; };
if (limit) if (limit)
#if !defined(ARCADIA_BUILD)
miniselect::floyd_rivest_partial_sort(res.begin(), res.begin() + limit, res.end(), less);
#else
std::partial_sort(res.begin(), res.begin() + limit, res.end(), less); std::partial_sort(res.begin(), res.begin() + limit, res.end(), less);
#endif
else else
std::sort(res.begin(), res.end(), less); std::sort(res.begin(), res.end(), less);
} }
@ -364,8 +372,11 @@ void ColumnString::updatePermutationImpl(size_t limit, Permutation & res, EqualR
return; return;
/// Since then we are working inside the interval. /// Since then we are working inside the interval.
#if !defined(ARCADIA_BUILD)
miniselect::floyd_rivest_partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, less);
#else
std::partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, less); std::partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, less);
#endif
size_t new_first = first; size_t new_first = first;
for (size_t j = first + 1; j < limit; ++j) for (size_t j = first + 1; j < limit; ++j)

7
src/Columns/ColumnTuple.cpp
View File

@ -9,6 +9,9 @@
#include <Common/assert_cast.h> #include <Common/assert_cast.h>
#include <Common/WeakHash.h> #include <Common/WeakHash.h>
#include <Core/Field.h> #include <Core/Field.h>
#if !defined(ARCADIA_BUILD)
#include <miniselect/floyd_rivest_select.h> // Y_IGNORE
#endif
namespace DB namespace DB
@ -352,7 +355,11 @@ void ColumnTuple::getPermutationImpl(size_t limit, Permutation & res, LessOperat
if (limit) if (limit)
{ {
#if !defined(ARCADIA_BUILD)
miniselect::floyd_rivest_partial_sort(res.begin(), res.begin() + limit, res.end(), less);
#else
std::partial_sort(res.begin(), res.begin() + limit, res.end(), less); std::partial_sort(res.begin(), res.begin() + limit, res.end(), less);
#endif
} }
else else
{ {

20
src/Columns/ColumnVector.cpp
View File

@ -17,7 +17,9 @@
#include <ext/bit_cast.h> #include <ext/bit_cast.h>
#include <ext/scope_guard.h> #include <ext/scope_guard.h>
#include <pdqsort.h> #include <pdqsort.h>
#if !defined(ARCADIA_BUILD)
#include <miniselect/floyd_rivest_select.h> // Y_IGNORE
#endif
#ifdef __SSE2__ #ifdef __SSE2__
#include <emmintrin.h> #include <emmintrin.h>
@ -156,9 +158,17 @@ void ColumnVector<T>::getPermutation(bool reverse, size_t limit, int nan_directi
res[i] = i; res[i] = i;
if (reverse) if (reverse)
#if !defined(ARCADIA_BUILD)
miniselect::floyd_rivest_partial_sort(res.begin(), res.begin() + limit, res.end(), greater(*this, nan_direction_hint));
#else
std::partial_sort(res.begin(), res.begin() + limit, res.end(), greater(*this, nan_direction_hint)); std::partial_sort(res.begin(), res.begin() + limit, res.end(), greater(*this, nan_direction_hint));
#endif
else else
#if !defined(ARCADIA_BUILD)
miniselect::floyd_rivest_partial_sort(res.begin(), res.begin() + limit, res.end(), less(*this, nan_direction_hint));
#else
std::partial_sort(res.begin(), res.begin() + limit, res.end(), less(*this, nan_direction_hint)); std::partial_sort(res.begin(), res.begin() + limit, res.end(), less(*this, nan_direction_hint));
#endif
} }
else else
{ {
@ -254,9 +264,17 @@ void ColumnVector<T>::updatePermutation(bool reverse, size_t limit, int nan_dire
/// Since then, we are working inside the interval. /// Since then, we are working inside the interval.
if (reverse) if (reverse)
#if !defined(ARCADIA_BUILD)
miniselect::floyd_rivest_partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, greater(*this, nan_direction_hint));
#else
std::partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, greater(*this, nan_direction_hint)); std::partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, greater(*this, nan_direction_hint));
#endif
else else
#if !defined(ARCADIA_BUILD)
miniselect::floyd_rivest_partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, less(*this, nan_direction_hint));
#else
std::partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, less(*this, nan_direction_hint)); std::partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, less(*this, nan_direction_hint));
#endif
size_t new_first = first; size_t new_first = first;
for (size_t j = first + 1; j < limit; ++j) for (size_t j = first + 1; j < limit; ++j)

2
src/Columns/tests/gtest_weak_hash_32.cpp
View File

@ -71,7 +71,7 @@ void checkColumn(
std::unordered_map<UInt32, T> map; std::unordered_map<UInt32, T> map;
size_t num_collisions = 0; size_t num_collisions = 0;
std::stringstream collisions_str; std::stringstream collisions_str; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
collisions_str.exceptions(std::ios::failbit); collisions_str.exceptions(std::ios::failbit);
for (size_t i = 0; i < eq_class.size(); ++i) for (size_t i = 0; i < eq_class.size(); ++i)

6
src/Common/Config/ConfigProcessor.cpp
View File

@ -5,7 +5,6 @@
#include <cstdlib> #include <cstdlib>
#include <cstring> #include <cstring>
#include <algorithm> #include <algorithm>
#include <sstream>
#include <functional> #include <functional>
#include <filesystem> #include <filesystem>
#include <Poco/DOM/Text.h> #include <Poco/DOM/Text.h>
@ -17,6 +16,8 @@
#include <Common/StringUtils/StringUtils.h> #include <Common/StringUtils/StringUtils.h>
#include <Common/Exception.h> #include <Common/Exception.h>
#include <common/getResource.h> #include <common/getResource.h>
#include <IO/WriteBufferFromString.h>
#include <IO/Operators.h>
#define PREPROCESSED_SUFFIX "-preprocessed" #define PREPROCESSED_SUFFIX "-preprocessed"
@ -537,8 +538,7 @@ XMLDocumentPtr ConfigProcessor::processConfig(
if (has_zk_includes) if (has_zk_includes)
*has_zk_includes = !contributing_zk_paths.empty(); *has_zk_includes = !contributing_zk_paths.empty();
std::stringstream comment; WriteBufferFromOwnString comment;
comment.exceptions(std::ios::failbit);
comment << " This file was generated automatically.\n"; comment << " This file was generated automatically.\n";
comment << " Do not edit it: it is likely to be discarded and generated again before it's read next time.\n"; comment << " Do not edit it: it is likely to be discarded and generated again before it's read next time.\n";
comment << " Files used to generate this file:"; comment << " Files used to generate this file:";

23
src/Common/ErrorCodes.cpp
View File

@ -519,34 +519,35 @@
M(550, CONDITIONAL_TREE_PARENT_NOT_FOUND) \ M(550, CONDITIONAL_TREE_PARENT_NOT_FOUND) \
M(551, ILLEGAL_PROJECTION_MANIPULATOR) \ M(551, ILLEGAL_PROJECTION_MANIPULATOR) \
M(552, UNRECOGNIZED_ARGUMENTS) \ M(552, UNRECOGNIZED_ARGUMENTS) \
\ M(553, LZMA_STREAM_ENCODER_FAILED) \
M(554, LZMA_STREAM_DECODER_FAILED) \
\
M(999, KEEPER_EXCEPTION) \ M(999, KEEPER_EXCEPTION) \
M(1000, POCO_EXCEPTION) \ M(1000, POCO_EXCEPTION) \
M(1001, STD_EXCEPTION) \ M(1001, STD_EXCEPTION) \
M(1002, UNKNOWN_EXCEPTION) \ M(1002, UNKNOWN_EXCEPTION)
/* See END */ /* See END */
namespace DB namespace DB
{ {
namespace ErrorCodes namespace ErrorCodes
{ {
#define M(VALUE, NAME) extern const Value NAME = VALUE; #define M(VALUE, NAME) extern const Value NAME = VALUE;
APPLY_FOR_ERROR_CODES(M) APPLY_FOR_ERROR_CODES(M)
#undef M #undef M
constexpr Value END = 3000; constexpr Value END = 3000;
std::atomic<Value> values[END + 1] {}; std::atomic<Value> values[END + 1]{};
struct ErrorCodesNames struct ErrorCodesNames
{ {
std::string_view names[END + 1]; std::string_view names[END + 1];
ErrorCodesNames() ErrorCodesNames()
{ {
#define M(VALUE, NAME) names[VALUE] = std::string_view(#NAME); #define M(VALUE, NAME) names[VALUE] = std::string_view(#NAME);
APPLY_FOR_ERROR_CODES(M) APPLY_FOR_ERROR_CODES(M)
#undef M #undef M
} }
} error_codes_names; } error_codes_names;
@ -557,7 +558,7 @@ namespace ErrorCodes
return error_codes_names.names[error_code]; return error_codes_names.names[error_code];
} }
ErrorCode end() { return END+1; } ErrorCode end() { return END + 1; }
} }
} }

6
src/Common/Exception.cpp
View File

@ -245,8 +245,7 @@ static std::string getExtraExceptionInfo(const std::exception & e)
std::string getCurrentExceptionMessage(bool with_stacktrace, bool check_embedded_stacktrace /*= false*/, bool with_extra_info /*= true*/) std::string getCurrentExceptionMessage(bool with_stacktrace, bool check_embedded_stacktrace /*= false*/, bool with_extra_info /*= true*/)
{ {
std::stringstream stream; WriteBufferFromOwnString stream;
stream.exceptions(std::ios::failbit);
try try
{ {
@ -365,8 +364,7 @@ void tryLogException(std::exception_ptr e, Poco::Logger * logger, const std::str
std::string getExceptionMessage(const Exception & e, bool with_stacktrace, bool check_embedded_stacktrace) std::string getExceptionMessage(const Exception & e, bool with_stacktrace, bool check_embedded_stacktrace)
{ {
std::stringstream stream; WriteBufferFromOwnString stream;
stream.exceptions(std::ios::failbit);
try try
{ {

4
src/Common/HTMLForm.h
View File

@ -16,13 +16,13 @@ struct HTMLForm : public Poco::Net::HTMLForm
HTMLForm(const Poco::Net::HTTPRequest & request) HTMLForm(const Poco::Net::HTTPRequest & request)
{ {
Poco::URI uri(request.getURI()); Poco::URI uri(request.getURI());
std::istringstream istr(uri.getRawQuery()); std::istringstream istr(uri.getRawQuery()); // STYLE_CHECK_ALLOW_STD_STRING_STREAM
readUrl(istr); readUrl(istr);
} }
HTMLForm(const Poco::URI & uri) HTMLForm(const Poco::URI & uri)
{ {
std::istringstream istr(uri.getRawQuery()); std::istringstream istr(uri.getRawQuery()); // STYLE_CHECK_ALLOW_STD_STRING_STREAM
readUrl(istr); readUrl(istr);
} }

32
src/Common/MemoryTracker.cpp
View File

@ -133,17 +133,13 @@ void MemoryTracker::alloc(Int64 size)
BlockerInThread untrack_lock; BlockerInThread untrack_lock;
ProfileEvents::increment(ProfileEvents::QueryMemoryLimitExceeded); ProfileEvents::increment(ProfileEvents::QueryMemoryLimitExceeded);
std::stringstream message; const auto * description = description_ptr.load(std::memory_order_relaxed);
message.exceptions(std::ios::failbit);
message << "Memory tracker";
if (const auto * description = description_ptr.load(std::memory_order_relaxed))
message << " " << description;
message << ": fault injected. Would use " << formatReadableSizeWithBinarySuffix(will_be)
<< " (attempt to allocate chunk of " << size << " bytes)"
<< ", maximum: " << formatReadableSizeWithBinarySuffix(current_hard_limit);
amount.fetch_sub(size, std::memory_order_relaxed); amount.fetch_sub(size, std::memory_order_relaxed);
throw DB::Exception(message.str(), DB::ErrorCodes::MEMORY_LIMIT_EXCEEDED); throw DB::Exception(DB::ErrorCodes::MEMORY_LIMIT_EXCEEDED,
"Memory tracker{}{}: fault injected. Would use {} (attempt to allocate chunk of {} bytes), maximum: {}",
description ? " " : "", description ? description : "",
formatReadableSizeWithBinarySuffix(will_be),
size, formatReadableSizeWithBinarySuffix(current_hard_limit));
} }
if (unlikely(current_profiler_limit && will_be > current_profiler_limit)) if (unlikely(current_profiler_limit && will_be > current_profiler_limit))
@ -166,17 +162,13 @@ void MemoryTracker::alloc(Int64 size)
BlockerInThread untrack_lock; BlockerInThread untrack_lock;
ProfileEvents::increment(ProfileEvents::QueryMemoryLimitExceeded); ProfileEvents::increment(ProfileEvents::QueryMemoryLimitExceeded);
std::stringstream message; const auto * description = description_ptr.load(std::memory_order_relaxed);
message.exceptions(std::ios::failbit);
message << "Memory limit";
if (const auto * description = description_ptr.load(std::memory_order_relaxed))
message << " " << description;
message << " exceeded: would use " << formatReadableSizeWithBinarySuffix(will_be)
<< " (attempt to allocate chunk of " << size << " bytes)"
<< ", maximum: " << formatReadableSizeWithBinarySuffix(current_hard_limit);
amount.fetch_sub(size, std::memory_order_relaxed); amount.fetch_sub(size, std::memory_order_relaxed);
throw DB::Exception(message.str(), DB::ErrorCodes::MEMORY_LIMIT_EXCEEDED); throw DB::Exception(DB::ErrorCodes::MEMORY_LIMIT_EXCEEDED,
"Memory limit{}{} exceeded: would use {} (attempt to allocate chunk of {} bytes), maximum: {}",
description ? " " : "", description ? description : "",
formatReadableSizeWithBinarySuffix(will_be),
size, formatReadableSizeWithBinarySuffix(current_hard_limit));
} }
updatePeak(will_be); updatePeak(will_be);

4
src/Common/ShellCommand.cpp
View File

@ -8,6 +8,7 @@
#include <common/logger_useful.h> #include <common/logger_useful.h>
#include <common/errnoToString.h> #include <common/errnoToString.h>
#include <IO/WriteHelpers.h> #include <IO/WriteHelpers.h>
#include <IO/Operators.h>
#include <unistd.h> #include <unistd.h>
#include <csignal> #include <csignal>
@ -73,8 +74,7 @@ ShellCommand::~ShellCommand()
void ShellCommand::logCommand(const char * filename, char * const argv[]) void ShellCommand::logCommand(const char * filename, char * const argv[])
{ {
std::stringstream args; WriteBufferFromOwnString args;
args.exceptions(std::ios::failbit);
for (int i = 0; argv != nullptr && argv[i] != nullptr; ++i) for (int i = 0; argv != nullptr && argv[i] != nullptr; ++i)
{ {
if (i > 0) if (i > 0)

8
src/Common/StackTrace.cpp
View File

@ -23,7 +23,7 @@
std::string signalToErrorMessage(int sig, const siginfo_t & info, const ucontext_t & context) std::string signalToErrorMessage(int sig, const siginfo_t & info, const ucontext_t & context)
{ {
std::stringstream error; std::stringstream error; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
error.exceptions(std::ios::failbit); error.exceptions(std::ios::failbit);
switch (sig) switch (sig)
{ {
@ -319,7 +319,7 @@ static void toStringEveryLineImpl(
const DB::SymbolIndex & symbol_index = DB::SymbolIndex::instance(); const DB::SymbolIndex & symbol_index = DB::SymbolIndex::instance();
std::unordered_map<std::string, DB::Dwarf> dwarfs; std::unordered_map<std::string, DB::Dwarf> dwarfs;
std::stringstream out; std::stringstream out; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
out.exceptions(std::ios::failbit); out.exceptions(std::ios::failbit);
for (size_t i = offset; i < size; ++i) for (size_t i = offset; i < size; ++i)
@ -359,7 +359,7 @@ static void toStringEveryLineImpl(
out.str({}); out.str({});
} }
#else #else
std::stringstream out; std::stringstream out; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
out.exceptions(std::ios::failbit); out.exceptions(std::ios::failbit);
for (size_t i = offset; i < size; ++i) for (size_t i = offset; i < size; ++i)
@ -375,7 +375,7 @@ static void toStringEveryLineImpl(
static std::string toStringImpl(const StackTrace::FramePointers & frame_pointers, size_t offset, size_t size) static std::string toStringImpl(const StackTrace::FramePointers & frame_pointers, size_t offset, size_t size)
{ {
std::stringstream out; std::stringstream out; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
out.exceptions(std::ios::failbit); out.exceptions(std::ios::failbit);
toStringEveryLineImpl(frame_pointers, offset, size, [&](const std::string & str) { out << str << '\n'; }); toStringEveryLineImpl(frame_pointers, offset, size, [&](const std::string & str) { out << str << '\n'; });
return out.str(); return out.str();

2
src/Common/StudentTTest.cpp
View File

@ -153,7 +153,7 @@ std::pair<bool, std::string> StudentTTest::compareAndReport(size_t confidence_le
double mean_confidence_interval = table_value * t_statistic; double mean_confidence_interval = table_value * t_statistic;
std::stringstream ss; std::stringstream ss; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
ss.exceptions(std::ios::failbit); ss.exceptions(std::ios::failbit);
if (mean_difference > mean_confidence_interval && (mean_difference - mean_confidence_interval > 0.0001)) /// difference must be more than 0.0001, to take into account connection latency. if (mean_difference > mean_confidence_interval && (mean_difference - mean_confidence_interval > 0.0001)) /// difference must be more than 0.0001, to take into account connection latency.

3
src/Common/ThreadProfileEvents.cpp
View File

@ -413,7 +413,8 @@ std::vector<size_t> PerfEventsCounters::eventIndicesFromString(const std::string
return result; return result;
} }
std::istringstream iss(events_list);
std::istringstream iss(events_list); // STYLE_CHECK_ALLOW_STD_STRING_STREAM
std::string event_name; std::string event_name;
while (std::getline(iss, event_name, ',')) while (std::getline(iss, event_name, ','))
{ {

10
src/Common/ThreadStatus.cpp
View File

@ -1,5 +1,3 @@
#include <sstream>
#include <Common/Exception.h> #include <Common/Exception.h>
#include <Common/ThreadProfileEvents.h> #include <Common/ThreadProfileEvents.h>
#include <Common/QueryProfiler.h> #include <Common/QueryProfiler.h>
@ -79,12 +77,10 @@ void ThreadStatus::assertState(const std::initializer_list<int> & permitted_stat
return; return;
} }
std::stringstream ss;
ss.exceptions(std::ios::failbit);
ss << "Unexpected thread state " << getCurrentState();
if (description) if (description)
ss << ": " << description; throw Exception(ErrorCodes::LOGICAL_ERROR, "Unexpected thread state {}: {}", getCurrentState(), description);
throw Exception(ss.str(), ErrorCodes::LOGICAL_ERROR); else
throw Exception(ErrorCodes::LOGICAL_ERROR, "Unexpected thread state {}", getCurrentState());
} }
void ThreadStatus::attachInternalTextLogsQueue(const InternalTextLogsQueuePtr & logs_queue, void ThreadStatus::attachInternalTextLogsQueue(const InternalTextLogsQueuePtr & logs_queue,

9
src/Common/UInt128.h
View File

@ -1,11 +1,11 @@
#pragma once #pragma once
#include <tuple> #include <tuple>
#include <sstream>
#include <iomanip> #include <iomanip>
#include <city.h> #include <city.h>
#include <Core/Types.h> #include <Core/Types.h>
#include <Common/hex.h>
#ifdef __SSE4_2__ #ifdef __SSE4_2__
#include <nmmintrin.h> #include <nmmintrin.h>
@ -48,10 +48,9 @@ struct UInt128
String toHexString() const String toHexString() const
{ {
std::ostringstream os; String res(2 * sizeof(UInt128), 0);
os.exceptions(std::ios::failbit); writeHexUIntLowercase(*this, res.data());
os << std::setw(16) << std::setfill('0') << std::hex << high << low; return res;
return String(os.str());
} }
bool inline operator== (const UInt128 rhs) const { return tuple() == rhs.tuple(); } bool inline operator== (const UInt128 rhs) const { return tuple() == rhs.tuple(); }

4
src/Common/XDBCBridgeHelper.h
View File

@ -1,6 +1,5 @@
#pragma once #pragma once
#include <sstream>
#include <IO/ReadHelpers.h> #include <IO/ReadHelpers.h>
#include <IO/ReadWriteBufferFromHTTP.h> #include <IO/ReadWriteBufferFromHTTP.h>
#include <Interpreters/Context.h> #include <Interpreters/Context.h>
@ -307,9 +306,6 @@ struct ODBCBridgeMixin
std::vector<std::string> cmd_args; std::vector<std::string> cmd_args;
path.setFileName("clickhouse-odbc-bridge"); path.setFileName("clickhouse-odbc-bridge");
std::stringstream command;
command.exceptions(std::ios::failbit);
#if !CLICKHOUSE_SPLIT_BINARY #if !CLICKHOUSE_SPLIT_BINARY
cmd_args.push_back("odbc-bridge"); cmd_args.push_back("odbc-bridge");
#endif #endif

2
src/Common/ZooKeeper/TestKeeper.cpp
View File

@ -218,7 +218,7 @@ std::pair<ResponsePtr, Undo> TestKeeperCreateRequest::process(TestKeeper::Contai
auto seq_num = it->second.seq_num; auto seq_num = it->second.seq_num;
++it->second.seq_num; ++it->second.seq_num;
std::stringstream seq_num_str; std::stringstream seq_num_str; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
seq_num_str.exceptions(std::ios::failbit); seq_num_str.exceptions(std::ios::failbit);
seq_num_str << std::setw(10) << std::setfill('0') << seq_num; seq_num_str << std::setw(10) << std::setfill('0') << seq_num;

12
src/Common/checkStackSize.cpp
View File

@ -3,7 +3,6 @@
#include <ext/scope_guard.h> #include <ext/scope_guard.h>
#include <pthread.h> #include <pthread.h>
#include <cstdint> #include <cstdint>
#include <sstream>
#if defined(__FreeBSD__) #if defined(__FreeBSD__)
# include <pthread_np.h> # include <pthread_np.h>
@ -80,13 +79,8 @@ __attribute__((__weak__)) void checkStackSize()
/// It's safe to assume that overflow in multiplying by two cannot occur. /// It's safe to assume that overflow in multiplying by two cannot occur.
if (stack_size * 2 > max_stack_size) if (stack_size * 2 > max_stack_size)
{ {
std::stringstream message; throw Exception(ErrorCodes::TOO_DEEP_RECURSION,
message.exceptions(std::ios::failbit); "Stack size too large. Stack address: {}, frame address: {}, stack size: {}, maximum stack size: {}",
message << "Stack size too large" stack_address, frame_address, stack_size, max_stack_size);
<< ". Stack address: " << stack_address
<< ", frame address: " << frame_address
<< ", stack size: " << stack_size
<< ", maximum stack size: " << max_stack_size;
throw Exception(message.str(), ErrorCodes::TOO_DEEP_RECURSION);
} }
} }

2
src/Common/formatReadable.cpp
View File

@ -1,6 +1,4 @@
#include <cmath> #include <cmath>
#include <sstream>
#include <iomanip>
#include <Common/formatReadable.h> #include <Common/formatReadable.h>
#include <IO/DoubleConverter.h> #include <IO/DoubleConverter.h>

33
src/Common/parseGlobs.cpp
View File

@ -1,4 +1,7 @@
#include <Common/parseGlobs.h> #include <Common/parseGlobs.h>
#include <IO/WriteBufferFromString.h>
#include <IO/ReadBufferFromString.h>
#include <IO/Operators.h>
#include <re2/re2.h> #include <re2/re2.h>
#include <re2/stringpiece.h> #include <re2/stringpiece.h>
#include <algorithm> #include <algorithm>
@ -18,21 +21,21 @@ namespace DB
*/ */
std::string makeRegexpPatternFromGlobs(const std::string & initial_str_with_globs) std::string makeRegexpPatternFromGlobs(const std::string & initial_str_with_globs)
{ {
std::ostringstream oss_for_escaping; /// FIXME make it better
oss_for_escaping.exceptions(std::ios::failbit); WriteBufferFromOwnString buf_for_escaping;
/// Escaping only characters that not used in glob syntax /// Escaping only characters that not used in glob syntax
for (const auto & letter : initial_str_with_globs) for (const auto & letter : initial_str_with_globs)
{ {
if ((letter == '[') || (letter == ']') || (letter == '|') || (letter == '+') || (letter == '-') || (letter == '(') || (letter == ')')) if ((letter == '[') || (letter == ']') || (letter == '|') || (letter == '+') || (letter == '-') || (letter == '(') || (letter == ')') || (letter == '\\'))
oss_for_escaping << '\\'; buf_for_escaping << '\\';
oss_for_escaping << letter; buf_for_escaping << letter;
} }
std::string escaped_with_globs = oss_for_escaping.str(); std::string escaped_with_globs = buf_for_escaping.str();
static const re2::RE2 enum_or_range(R"({([\d]+\.\.[\d]+|[^{}*,]+,[^{}*]*[^{}*,])})"); /// regexp for {expr1,expr2,expr3} or {M..N}, where M and N - non-negative integers, expr's should be without {}*, static const re2::RE2 enum_or_range(R"({([\d]+\.\.[\d]+|[^{}*,]+,[^{}*]*[^{}*,])})"); /// regexp for {expr1,expr2,expr3} or {M..N}, where M and N - non-negative integers, expr's should be without {}*,
re2::StringPiece input(escaped_with_globs); re2::StringPiece input(escaped_with_globs);
re2::StringPiece matched; re2::StringPiece matched;
std::ostringstream oss_for_replacing; std::ostringstream oss_for_replacing; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
oss_for_replacing.exceptions(std::ios::failbit); oss_for_replacing.exceptions(std::ios::failbit);
size_t current_index = 0; size_t current_index = 0;
while (RE2::FindAndConsume(&input, enum_or_range, &matched)) while (RE2::FindAndConsume(&input, enum_or_range, &matched))
@ -45,9 +48,8 @@ std::string makeRegexpPatternFromGlobs(const std::string & initial_str_with_glob
size_t range_begin = 0; size_t range_begin = 0;
size_t range_end = 0; size_t range_end = 0;
char point; char point;
std::istringstream iss_range(buffer); ReadBufferFromString buf_range(buffer);
iss_range.exceptions(std::ios::failbit); buf_range >> range_begin >> point >> point >> range_end;
iss_range >> range_begin >> point >> point >> range_end;
bool leading_zeros = buffer[0] == '0'; bool leading_zeros = buffer[0] == '0';
size_t num_len = std::to_string(range_end).size(); size_t num_len = std::to_string(range_end).size();
if (leading_zeros) if (leading_zeros)
@ -71,20 +73,19 @@ std::string makeRegexpPatternFromGlobs(const std::string & initial_str_with_glob
} }
oss_for_replacing << escaped_with_globs.substr(current_index); oss_for_replacing << escaped_with_globs.substr(current_index);
std::string almost_res = oss_for_replacing.str(); std::string almost_res = oss_for_replacing.str();
std::ostringstream oss_final_processing; WriteBufferFromOwnString buf_final_processing;
oss_final_processing.exceptions(std::ios::failbit);
for (const auto & letter : almost_res) for (const auto & letter : almost_res)
{ {
if ((letter == '?') || (letter == '*')) if ((letter == '?') || (letter == '*'))
{ {
oss_final_processing << "[^/]"; /// '?' is any symbol except '/' buf_final_processing << "[^/]"; /// '?' is any symbol except '/'
if (letter == '?') if (letter == '?')
continue; continue;
} }
if ((letter == '.') || (letter == '{') || (letter == '}')) if ((letter == '.') || (letter == '{') || (letter == '}'))
oss_final_processing << '\\'; buf_final_processing << '\\';
oss_final_processing << letter; buf_final_processing << letter;
} }
return oss_final_processing.str(); return buf_final_processing.str();
} }
} }

3
src/Common/tests/gtest_getMultipleValuesFromConfig.cpp
View File

@ -9,7 +9,8 @@ using namespace DB;
TEST(Common, getMultipleValuesFromConfig) TEST(Common, getMultipleValuesFromConfig)
{ {
std::istringstream xml_isteam(R"END(<?xml version="1.0"?> std::istringstream // STYLE_CHECK_ALLOW_STD_STRING_STREAM
xml_isteam(R"END(<?xml version="1.0"?>
<yandex> <yandex>
<first_level> <first_level>
<second_level>0</second_level> <second_level>0</second_level>

12
src/Common/tests/gtest_sensitive_data_masker.cpp
View File

@ -102,7 +102,8 @@ TEST(Common, SensitiveDataMasker)
EXPECT_EQ(maskerbad.wipeSensitiveData(x), 0); EXPECT_EQ(maskerbad.wipeSensitiveData(x), 0);
{ {
std::istringstream xml_isteam(R"END(<?xml version="1.0"?> std::istringstream // STYLE_CHECK_ALLOW_STD_STRING_STREAM
xml_isteam(R"END(<?xml version="1.0"?>
<clickhouse> <clickhouse>
<query_masking_rules> <query_masking_rules>
<rule> <rule>
@ -152,7 +153,8 @@ TEST(Common, SensitiveDataMasker)
try try
{ {
std::istringstream xml_isteam_bad(R"END(<?xml version="1.0"?> std::istringstream // STYLE_CHECK_ALLOW_STD_STRING_STREAM
xml_isteam_bad(R"END(<?xml version="1.0"?>
<clickhouse> <clickhouse>
<query_masking_rules> <query_masking_rules>
<rule> <rule>
@ -181,7 +183,8 @@ TEST(Common, SensitiveDataMasker)
try try
{ {
std::istringstream xml_isteam_bad(R"END(<?xml version="1.0"?> std::istringstream // STYLE_CHECK_ALLOW_STD_STRING_STREAM
xml_isteam_bad(R"END(<?xml version="1.0"?>
<clickhouse> <clickhouse>
<query_masking_rules> <query_masking_rules>
<rule><name>test</name></rule> <rule><name>test</name></rule>
@ -203,7 +206,8 @@ TEST(Common, SensitiveDataMasker)
try try
{ {
std::istringstream xml_isteam_bad(R"END(<?xml version="1.0"?> std::istringstream // STYLE_CHECK_ALLOW_STD_STRING_STREAM
xml_isteam_bad(R"END(<?xml version="1.0"?>
<clickhouse> <clickhouse>
<query_masking_rules> <query_masking_rules>
<rule><name>test</name><regexp>())(</regexp></rule> <rule><name>test</name><regexp>())(</regexp></rule>

14
src/Compression/CompressedReadBufferBase.cpp
View File

@ -12,6 +12,7 @@
#include <IO/BufferWithOwnMemory.h> #include <IO/BufferWithOwnMemory.h>
#include <Compression/CompressionInfo.h> #include <Compression/CompressionInfo.h>
#include <IO/WriteHelpers.h> #include <IO/WriteHelpers.h>
#include <IO/Operators.h>
namespace ProfileEvents namespace ProfileEvents
@ -42,7 +43,7 @@ static void validateChecksum(char * data, size_t size, const Checksum expected_c
if (expected_checksum == calculated_checksum) if (expected_checksum == calculated_checksum)
return; return;
std::stringstream message; WriteBufferFromOwnString message;
/// TODO mess up of endianness in error message. /// TODO mess up of endianness in error message.
message << "Checksum doesn't match: corrupted data." message << "Checksum doesn't match: corrupted data."
@ -50,7 +51,16 @@ static void validateChecksum(char * data, size_t size, const Checksum expected_c
+ ". Actual: " + getHexUIntLowercase(calculated_checksum.first) + getHexUIntLowercase(calculated_checksum.second) + ". Actual: " + getHexUIntLowercase(calculated_checksum.first) + getHexUIntLowercase(calculated_checksum.second)
+ ". Size of compressed block: " + toString(size); + ". Size of compressed block: " + toString(size);
const char * message_hardware_failure = "This is most likely due to hardware failure. If you receive broken data over network and the error does not repeat every time, this can be caused by bad RAM on network interface controller or bad controller itself or bad RAM on network switches or bad CPU on network switches (look at the logs on related network switches; note that TCP checksums don't help) or bad RAM on host (look at dmesg or kern.log for enormous amount of EDAC errors, ECC-related reports, Machine Check Exceptions, mcelog; note that ECC memory can fail if the number of errors is huge) or bad CPU on host. If you read data from disk, this can be caused by disk bit rott. This exception protects ClickHouse from data corruption due to hardware failures."; const char * message_hardware_failure = "This is most likely due to hardware failure. "
"If you receive broken data over network and the error does not repeat every time, "
"this can be caused by bad RAM on network interface controller or bad controller itself "
"or bad RAM on network switches or bad CPU on network switches "
"(look at the logs on related network switches; note that TCP checksums don't help) "
"or bad RAM on host (look at dmesg or kern.log for enormous amount of EDAC errors, "
"ECC-related reports, Machine Check Exceptions, mcelog; note that ECC memory can fail "
"if the number of errors is huge) or bad CPU on host. If you read data from disk, "
"this can be caused by disk bit rott. This exception protects ClickHouse "
"from data corruption due to hardware failures.";
auto flip_bit = [](char * buf, size_t pos) auto flip_bit = [](char * buf, size_t pos)
{ {

5
src/Compression/tests/compressed_buffer.cpp
View File

@ -51,10 +51,7 @@ int main(int, char **)
if (x != i) if (x != i)
{ {
std::stringstream s; throw DB::Exception(0, "Failed!, read: {}, expected: {}", x, i);
s.exceptions(std::ios::failbit);
s << "Failed!, read: " << x << ", expected: " << i;
throw DB::Exception(s.str(), 0);
} }
} }
stopwatch.stop(); stopwatch.stop();

8
src/Core/MySQL/IMySQLReadPacket.cpp
View File

@ -1,5 +1,4 @@
#include <Core/MySQL/IMySQLReadPacket.h> #include <Core/MySQL/IMySQLReadPacket.h>
#include <sstream>
#include <IO/MySQLPacketPayloadReadBuffer.h> #include <IO/MySQLPacketPayloadReadBuffer.h>
#include <IO/LimitReadBuffer.h> #include <IO/LimitReadBuffer.h>
@ -21,10 +20,9 @@ void IMySQLReadPacket::readPayload(ReadBuffer & in, uint8_t & sequence_id)
readPayloadImpl(payload); readPayloadImpl(payload);
if (!payload.eof()) if (!payload.eof())
{ {
std::stringstream tmp; throw Exception(ErrorCodes::UNKNOWN_PACKET_FROM_CLIENT,
tmp.exceptions(std::ios::failbit); "Packet payload is not fully read. Stopped after {} bytes, while {} bytes are in buffer.",
tmp << "Packet payload is not fully read. Stopped after " << payload.count() << " bytes, while " << payload.available() << " bytes are in buffer."; payload.count(), payload.available());
throw Exception(tmp.str(), ErrorCodes::UNKNOWN_PACKET_FROM_CLIENT);
} }
} }

12
src/Core/MySQL/IMySQLWritePacket.cpp
View File

@ -1,10 +1,14 @@
#include <Core/MySQL/IMySQLWritePacket.h> #include <Core/MySQL/IMySQLWritePacket.h>
#include <IO/MySQLPacketPayloadWriteBuffer.h> #include <IO/MySQLPacketPayloadWriteBuffer.h>
#include <sstream>
namespace DB namespace DB
{ {
namespace ErrorCodes
{
extern const int LOGICAL_ERROR;
}
namespace MySQLProtocol namespace MySQLProtocol
{ {
@ -15,10 +19,8 @@ void IMySQLWritePacket::writePayload(WriteBuffer & buffer, uint8_t & sequence_id
buf.next(); buf.next();
if (buf.remainingPayloadSize()) if (buf.remainingPayloadSize())
{ {
std::stringstream ss; throw Exception(ErrorCodes::LOGICAL_ERROR, "Incomplete payload. Written {} bytes, expected {} bytes.",
ss.exceptions(std::ios::failbit); getPayloadSize() - buf.remainingPayloadSize(), getPayloadSize());
ss << "Incomplete payload. Written " << getPayloadSize() - buf.remainingPayloadSize() << " bytes, expected " << getPayloadSize() << " bytes.";
throw Exception(ss.str(), 0);
} }
} }

101
src/Core/MySQL/MySQLReplication.cpp
View File

@ -4,6 +4,7 @@
#include <IO/ReadBufferFromString.h> #include <IO/ReadBufferFromString.h>
#include <IO/MySQLBinlogEventReadBuffer.h> #include <IO/MySQLBinlogEventReadBuffer.h>
#include <IO/ReadHelpers.h> #include <IO/ReadHelpers.h>
#include <IO/Operators.h>
#include <common/DateLUT.h> #include <common/DateLUT.h>
#include <Common/FieldVisitors.h> #include <Common/FieldVisitors.h>
#include <Core/MySQL/PacketsGeneric.h> #include <Core/MySQL/PacketsGeneric.h>
@ -35,15 +36,15 @@ namespace MySQLReplication
payload.readStrict(reinterpret_cast<char *>(&flags), 2); payload.readStrict(reinterpret_cast<char *>(&flags), 2);
} }
void EventHeader::dump(std::ostream & out) const void EventHeader::dump(WriteBuffer & out) const
{ {
out << "\n=== " << to_string(this->type) << " ===" << std::endl; out << "\n=== " << to_string(this->type) << " ===" << '\n';
out << "Timestamp: " << this->timestamp << std::endl; out << "Timestamp: " << this->timestamp << '\n';
out << "Event Type: " << this->type << std::endl; out << "Event Type: " << to_string(this->type) << '\n';
out << "Server ID: " << this->server_id << std::endl; out << "Server ID: " << this->server_id << '\n';
out << "Event Size: " << this->event_size << std::endl; out << "Event Size: " << this->event_size << '\n';
out << "Log Pos: " << this->log_pos << std::endl; out << "Log Pos: " << this->log_pos << '\n';
out << "Flags: " << this->flags << std::endl; out << "Flags: " << this->flags << '\n';
} }
/// https://dev.mysql.com/doc/internals/en/format-description-event.html /// https://dev.mysql.com/doc/internals/en/format-description-event.html
@ -60,13 +61,13 @@ namespace MySQLReplication
readStringUntilEOF(event_type_header_length, payload); readStringUntilEOF(event_type_header_length, payload);
} }
void FormatDescriptionEvent::dump(std::ostream & out) const void FormatDescriptionEvent::dump(WriteBuffer & out) const
{ {
header.dump(out); header.dump(out);
out << "Binlog Version: " << this->binlog_version << std::endl; out << "Binlog Version: " << this->binlog_version << '\n';
out << "Server Version: " << this->server_version << std::endl; out << "Server Version: " << this->server_version << '\n';
out << "Create Timestamp: " << this->create_timestamp << std::endl; out << "Create Timestamp: " << this->create_timestamp << '\n';
out << "Event Header Len: " << std::to_string(this->event_header_length) << std::endl; out << "Event Header Len: " << std::to_string(this->event_header_length) << '\n';
} }
/// https://dev.mysql.com/doc/internals/en/rotate-event.html /// https://dev.mysql.com/doc/internals/en/rotate-event.html
@ -76,11 +77,11 @@ namespace MySQLReplication
readStringUntilEOF(next_binlog, payload); readStringUntilEOF(next_binlog, payload);
} }
void RotateEvent::dump(std::ostream & out) const void RotateEvent::dump(WriteBuffer & out) const
{ {
header.dump(out); header.dump(out);
out << "Position: " << this->position << std::endl; out << "Position: " << this->position << '\n';
out << "Next Binlog: " << this->next_binlog << std::endl; out << "Next Binlog: " << this->next_binlog << '\n';
} }
/// https://dev.mysql.com/doc/internals/en/query-event.html /// https://dev.mysql.com/doc/internals/en/query-event.html
@ -116,24 +117,24 @@ namespace MySQLReplication
} }
} }
void QueryEvent::dump(std::ostream & out) const void QueryEvent::dump(WriteBuffer & out) const
{ {
header.dump(out); header.dump(out);
out << "Thread ID: " << this->thread_id << std::endl; out << "Thread ID: " << this->thread_id << '\n';
out << "Execution Time: " << this->exec_time << std::endl; out << "Execution Time: " << this->exec_time << '\n';
out << "Schema Len: " << std::to_string(this->schema_len) << std::endl; out << "Schema Len: " << std::to_string(this->schema_len) << '\n';
out << "Error Code: " << this->error_code << std::endl; out << "Error Code: " << this->error_code << '\n';
out << "Status Len: " << this->status_len << std::endl; out << "Status Len: " << this->status_len << '\n';
out << "Schema: " << this->schema << std::endl; out << "Schema: " << this->schema << '\n';
out << "Query: " << this->query << std::endl; out << "Query: " << this->query << '\n';
} }
void XIDEvent::parseImpl(ReadBuffer & payload) { payload.readStrict(reinterpret_cast<char *>(&xid), 8); } void XIDEvent::parseImpl(ReadBuffer & payload) { payload.readStrict(reinterpret_cast<char *>(&xid), 8); }
void XIDEvent::dump(std::ostream & out) const void XIDEvent::dump(WriteBuffer & out) const
{ {
header.dump(out); header.dump(out);
out << "XID: " << this->xid << std::endl; out << "XID: " << this->xid << '\n';
} }
void TableMapEvent::parseImpl(ReadBuffer & payload) void TableMapEvent::parseImpl(ReadBuffer & payload)
@ -238,21 +239,23 @@ namespace MySQLReplication
} }
} }
void TableMapEvent::dump(std::ostream & out) const void TableMapEvent::dump(WriteBuffer & out) const
{ {
header.dump(out); header.dump(out);
out << "Table ID: " << this->table_id << std::endl; out << "Table ID: " << this->table_id << '\n';
out << "Flags: " << this->flags << std::endl; out << "Flags: " << this->flags << '\n';
out << "Schema Len: " << std::to_string(this->schema_len) << std::endl; out << "Schema Len: " << std::to_string(this->schema_len) << '\n';
out << "Schema: " << this->schema << std::endl; out << "Schema: " << this->schema << '\n';
out << "Table Len: " << std::to_string(this->table_len) << std::endl; out << "Table Len: " << std::to_string(this->table_len) << '\n';
out << "Table: " << this->table << std::endl; out << "Table: " << this->table << '\n';
out << "Column Count: " << this->column_count << std::endl; out << "Column Count: " << this->column_count << '\n';
for (auto i = 0U; i < column_count; i++) for (auto i = 0U; i < column_count; i++)
{ {
out << "Column Type [" << i << "]: " << std::to_string(column_type[i]) << ", Meta: " << column_meta[i] << std::endl; out << "Column Type [" << i << "]: " << std::to_string(column_type[i]) << ", Meta: " << column_meta[i] << '\n';
} }
out << "Null Bitmap: " << this->null_bitmap << std::endl; String bitmap_str;
boost::to_string(this->null_bitmap, bitmap_str);
out << "Null Bitmap: " << bitmap_str << '\n';
} }
void RowsEvent::parseImpl(ReadBuffer & payload) void RowsEvent::parseImpl(ReadBuffer & payload)
@ -631,16 +634,16 @@ namespace MySQLReplication
rows.push_back(row); rows.push_back(row);
} }
void RowsEvent::dump(std::ostream & out) const void RowsEvent::dump(WriteBuffer & out) const
{ {
FieldVisitorToString to_string; FieldVisitorToString to_string;
header.dump(out); header.dump(out);
out << "Schema: " << this->schema << std::endl; out << "Schema: " << this->schema << '\n';
out << "Table: " << this->table << std::endl; out << "Table: " << this->table << '\n';
for (auto i = 0U; i < rows.size(); i++) for (auto i = 0U; i < rows.size(); i++)
{ {
out << "Row[" << i << "]: " << applyVisitor(to_string, rows[i]) << std::endl; out << "Row[" << i << "]: " << applyVisitor(to_string, rows[i]) << '\n';
} }
} }
@ -663,22 +666,22 @@ namespace MySQLReplication
payload.ignoreAll(); payload.ignoreAll();
} }
void GTIDEvent::dump(std::ostream & out) const void GTIDEvent::dump(WriteBuffer & out) const
{ {
WriteBufferFromOwnString ws; WriteBufferFromOwnString ws;
writeUUIDText(gtid.uuid, ws); writeUUIDText(gtid.uuid, ws);
auto gtid_next = ws.str() + ":" + std::to_string(gtid.seq_no); auto gtid_next = ws.str() + ":" + std::to_string(gtid.seq_no);
header.dump(out); header.dump(out);
out << "GTID Next: " << gtid_next << std::endl; out << "GTID Next: " << gtid_next << '\n';
} }
void DryRunEvent::parseImpl(ReadBuffer & payload) { payload.ignoreAll(); } void DryRunEvent::parseImpl(ReadBuffer & payload) { payload.ignoreAll(); }
void DryRunEvent::dump(std::ostream & out) const void DryRunEvent::dump(WriteBuffer & out) const
{ {
header.dump(out); header.dump(out);
out << "[DryRun Event]" << std::endl; out << "[DryRun Event]" << '\n';
} }
/// Update binlog name/position/gtid based on the event type. /// Update binlog name/position/gtid based on the event type.
@ -716,12 +719,12 @@ namespace MySQLReplication
gtid_sets.parse(gtid_sets_); gtid_sets.parse(gtid_sets_);
} }
void Position::dump(std::ostream & out) const void Position::dump(WriteBuffer & out) const
{ {
out << "\n=== Binlog Position ===" << std::endl; out << "\n=== Binlog Position ===" << '\n';
out << "Binlog: " << this->binlog_name << std::endl; out << "Binlog: " << this->binlog_name << '\n';
out << "Position: " << this->binlog_pos << std::endl; out << "Position: " << this->binlog_pos << '\n';
out << "GTIDSets: " << this->gtid_sets.toString() << std::endl; out << "GTIDSets: " << this->gtid_sets.toString() << '\n';
} }
void MySQLFlavor::readPayloadImpl(ReadBuffer & payload) void MySQLFlavor::readPayloadImpl(ReadBuffer & payload)

22
src/Core/MySQL/MySQLReplication.h
View File

@ -309,7 +309,7 @@ namespace MySQLReplication
UInt16 flags; UInt16 flags;
EventHeader() : timestamp(0), server_id(0), event_size(0), log_pos(0), flags(0) { } EventHeader() : timestamp(0), server_id(0), event_size(0), log_pos(0), flags(0) { }
void dump(std::ostream & out) const; void dump(WriteBuffer & out) const;
void parse(ReadBuffer & payload); void parse(ReadBuffer & payload);
}; };
@ -321,7 +321,7 @@ namespace MySQLReplication
EventBase(EventHeader && header_) : header(std::move(header_)) {} EventBase(EventHeader && header_) : header(std::move(header_)) {}
virtual ~EventBase() = default; virtual ~EventBase() = default;
virtual void dump(std::ostream & out) const = 0; virtual void dump(WriteBuffer & out) const = 0;
virtual void parseEvent(ReadBuffer & payload) { parseImpl(payload); } virtual void parseEvent(ReadBuffer & payload) { parseImpl(payload); }
virtual MySQLEventType type() const { return MYSQL_UNHANDLED_EVENT; } virtual MySQLEventType type() const { return MYSQL_UNHANDLED_EVENT; }
@ -344,7 +344,7 @@ namespace MySQLReplication
UInt8 event_header_length; UInt8 event_header_length;
String event_type_header_length; String event_type_header_length;
void dump(std::ostream & out) const override; void dump(WriteBuffer & out) const override;
void parseImpl(ReadBuffer & payload) override; void parseImpl(ReadBuffer & payload) override;
private: private:
@ -358,7 +358,7 @@ namespace MySQLReplication
String next_binlog; String next_binlog;
RotateEvent(EventHeader && header_) : EventBase(std::move(header_)), position(0) {} RotateEvent(EventHeader && header_) : EventBase(std::move(header_)), position(0) {}
void dump(std::ostream & out) const override; void dump(WriteBuffer & out) const override;
protected: protected:
void parseImpl(ReadBuffer & payload) override; void parseImpl(ReadBuffer & payload) override;
@ -389,7 +389,7 @@ namespace MySQLReplication
{ {
} }
void dump(std::ostream & out) const override; void dump(WriteBuffer & out) const override;
MySQLEventType type() const override { return MYSQL_QUERY_EVENT; } MySQLEventType type() const override { return MYSQL_QUERY_EVENT; }
protected: protected:
@ -404,7 +404,7 @@ namespace MySQLReplication
protected: protected:
UInt64 xid; UInt64 xid;
void dump(std::ostream & out) const override; void dump(WriteBuffer & out) const override;
void parseImpl(ReadBuffer & payload) override; void parseImpl(ReadBuffer & payload) override;
}; };
@ -423,7 +423,7 @@ namespace MySQLReplication
Bitmap null_bitmap; Bitmap null_bitmap;
TableMapEvent(EventHeader && header_) : EventBase(std::move(header_)), table_id(0), flags(0), schema_len(0), table_len(0), column_count(0) {} TableMapEvent(EventHeader && header_) : EventBase(std::move(header_)), table_id(0), flags(0), schema_len(0), table_len(0), column_count(0) {}
void dump(std::ostream & out) const override; void dump(WriteBuffer & out) const override;
protected: protected:
void parseImpl(ReadBuffer & payload) override; void parseImpl(ReadBuffer & payload) override;
@ -445,7 +445,7 @@ namespace MySQLReplication
table = table_map->table; table = table_map->table;
} }
void dump(std::ostream & out) const override; void dump(WriteBuffer & out) const override;
protected: protected:
UInt64 table_id; UInt64 table_id;
@ -489,7 +489,7 @@ namespace MySQLReplication
GTID gtid; GTID gtid;
GTIDEvent(EventHeader && header_) : EventBase(std::move(header_)), commit_flag(0) {} GTIDEvent(EventHeader && header_) : EventBase(std::move(header_)), commit_flag(0) {}
void dump(std::ostream & out) const override; void dump(WriteBuffer & out) const override;
protected: protected:
void parseImpl(ReadBuffer & payload) override; void parseImpl(ReadBuffer & payload) override;
@ -499,7 +499,7 @@ namespace MySQLReplication
{ {
public: public:
DryRunEvent(EventHeader && header_) : EventBase(std::move(header_)) {} DryRunEvent(EventHeader && header_) : EventBase(std::move(header_)) {}
void dump(std::ostream & out) const override; void dump(WriteBuffer & out) const override;
protected: protected:
void parseImpl(ReadBuffer & payload) override; void parseImpl(ReadBuffer & payload) override;
@ -515,7 +515,7 @@ namespace MySQLReplication
Position() : binlog_pos(0) { } Position() : binlog_pos(0) { }
void update(BinlogEventPtr event); void update(BinlogEventPtr event);
void update(UInt64 binlog_pos_, const String & binlog_name_, const String & gtid_sets_); void update(UInt64 binlog_pos_, const String & binlog_name_, const String & gtid_sets_);
void dump(std::ostream & out) const; void dump(WriteBuffer & out) const;
}; };
class IFlavor : public MySQLProtocol::IMySQLReadPacket class IFlavor : public MySQLProtocol::IMySQLReadPacket

1
src/Core/Settings.h
View File

@ -400,6 +400,7 @@ class IColumn;
\ \
/** Obsolete settings that do nothing but left for compatibility reasons. Remove each one after half a year of obsolescence. */ \ /** Obsolete settings that do nothing but left for compatibility reasons. Remove each one after half a year of obsolescence. */ \
\ \
M(UInt64, max_memory_usage_for_all_queries, 0, "Obsolete. Will be removed after 2020-10-20", 0) \
M(UInt64, multiple_joins_rewriter_version, 0, "Obsolete setting, does nothing. Will be removed after 2021-03-31", 0) \ M(UInt64, multiple_joins_rewriter_version, 0, "Obsolete setting, does nothing. Will be removed after 2021-03-31", 0) \
M(Bool, experimental_use_processors, true, "Obsolete setting, does nothing. Will be removed after 2020-11-29.", 0) \ M(Bool, experimental_use_processors, true, "Obsolete setting, does nothing. Will be removed after 2020-11-29.", 0) \
M(Bool, force_optimize_skip_unused_shards_no_nested, false, "Obsolete setting, does nothing. Will be removed after 2020-12-01. Use force_optimize_skip_unused_shards_nesting instead.", 0) \ M(Bool, force_optimize_skip_unused_shards_no_nested, false, "Obsolete setting, does nothing. Will be removed after 2020-12-01. Use force_optimize_skip_unused_shards_nesting instead.", 0) \

5
src/Core/SortDescription.h
View File

@ -60,10 +60,7 @@ struct SortColumnDescription
std::string dump() const std::string dump() const
{ {
std::stringstream ss; return fmt::format("{}:{}:dir {}nulls ", column_name, column_number, direction, nulls_direction);
ss.exceptions(std::ios::failbit);
ss << column_name << ":" << column_number << ":dir " << direction << "nulls " << nulls_direction;
return ss.str();
} }
}; };

21
src/Core/tests/mysql_protocol.cpp
View File

@ -7,6 +7,7 @@
#include <Core/MySQL/PacketsProtocolText.h> #include <Core/MySQL/PacketsProtocolText.h>
#include <IO/ReadBufferFromString.h> #include <IO/ReadBufferFromString.h>
#include <IO/WriteBufferFromString.h> #include <IO/WriteBufferFromString.h>
#include <IO/WriteBufferFromOStream.h>
#include <boost/program_options.hpp> #include <boost/program_options.hpp>
@ -329,6 +330,8 @@ int main(int argc, char ** argv)
slave.connect(); slave.connect();
slave.startBinlogDumpGTID(slave_id, replicate_db, gtid_sets); slave.startBinlogDumpGTID(slave_id, replicate_db, gtid_sets);
WriteBufferFromOStream cerr(std::cerr);
/// Read one binlog event on by one. /// Read one binlog event on by one.
while (true) while (true)
{ {
@ -337,40 +340,40 @@ int main(int argc, char ** argv)
{ {
case MYSQL_QUERY_EVENT: { case MYSQL_QUERY_EVENT: {
auto binlog_event = std::static_pointer_cast<QueryEvent>(event); auto binlog_event = std::static_pointer_cast<QueryEvent>(event);
binlog_event->dump(std::cerr); binlog_event->dump(cerr);
Position pos = slave.getPosition(); Position pos = slave.getPosition();
pos.dump(std::cerr); pos.dump(cerr);
break; break;
} }
case MYSQL_WRITE_ROWS_EVENT: { case MYSQL_WRITE_ROWS_EVENT: {
auto binlog_event = std::static_pointer_cast<WriteRowsEvent>(event); auto binlog_event = std::static_pointer_cast<WriteRowsEvent>(event);
binlog_event->dump(std::cerr); binlog_event->dump(cerr);
Position pos = slave.getPosition(); Position pos = slave.getPosition();
pos.dump(std::cerr); pos.dump(cerr);
break; break;
} }
case MYSQL_UPDATE_ROWS_EVENT: { case MYSQL_UPDATE_ROWS_EVENT: {
auto binlog_event = std::static_pointer_cast<UpdateRowsEvent>(event); auto binlog_event = std::static_pointer_cast<UpdateRowsEvent>(event);
binlog_event->dump(std::cerr); binlog_event->dump(cerr);
Position pos = slave.getPosition(); Position pos = slave.getPosition();
pos.dump(std::cerr); pos.dump(cerr);
break; break;
} }
case MYSQL_DELETE_ROWS_EVENT: { case MYSQL_DELETE_ROWS_EVENT: {
auto binlog_event = std::static_pointer_cast<DeleteRowsEvent>(event); auto binlog_event = std::static_pointer_cast<DeleteRowsEvent>(event);
binlog_event->dump(std::cerr); binlog_event->dump(cerr);
Position pos = slave.getPosition(); Position pos = slave.getPosition();
pos.dump(std::cerr); pos.dump(cerr);
break; break;
} }
default: default:
if (event->header.type != MySQLReplication::EventType::HEARTBEAT_EVENT) if (event->header.type != MySQLReplication::EventType::HEARTBEAT_EVENT)
{ {
event->dump(std::cerr); event->dump(cerr);
} }
break; break;
} }

38
src/DataStreams/CheckConstraintsBlockOutputStream.cpp
View File

@ -59,15 +59,10 @@ void CheckConstraintsBlockOutputStream::write(const Block & block)
/// Is violated. /// Is violated.
if (!value) if (!value)
{ {
std::stringstream exception_message; throw Exception(ErrorCodes::VIOLATED_CONSTRAINT,
exception_message.exceptions(std::ios::failbit); "Constraint {} for table {} is violated, because it is a constant expression returning 0. "
"It is most likely an error in table definition.",
exception_message << "Constraint " << backQuote(constraint_ptr->name) backQuote(constraint_ptr->name), table_id.getNameForLogs());
<< " for table " << table_id.getNameForLogs()
<< " is violated, because it is a constant expression returning 0."
<< " It is most likely an error in table definition.";
throw Exception{exception_message.str(), ErrorCodes::VIOLATED_CONSTRAINT};
} }
} }
else else
@ -87,28 +82,27 @@ void CheckConstraintsBlockOutputStream::write(const Block & block)
Names related_columns = constraint_expr->getRequiredColumns(); Names related_columns = constraint_expr->getRequiredColumns();
std::stringstream exception_message;
exception_message.exceptions(std::ios::failbit);
exception_message << "Constraint " << backQuote(constraint_ptr->name)
<< " for table " << table_id.getNameForLogs()
<< " is violated at row " << (rows_written + row_idx + 1)
<< ". Expression: (" << serializeAST(*(constraint_ptr->expr), true) << ")"
<< ". Column values";
bool first = true; bool first = true;
String column_values_msg;
constexpr size_t approx_bytes_for_col = 32;
column_values_msg.reserve(approx_bytes_for_col * related_columns.size());
for (const auto & name : related_columns) for (const auto & name : related_columns)
{ {
const IColumn & column = *block.getByName(name).column; const IColumn & column = *block.getByName(name).column;
assert(row_idx < column.size()); assert(row_idx < column.size());
exception_message << (first ? ": " : ", ") if (!first)
<< backQuoteIfNeed(name) << " = " << applyVisitor(FieldVisitorToString(), column[row_idx]); column_values_msg.append(", ");
column_values_msg.append(backQuoteIfNeed(name));
column_values_msg.append(" = ");
column_values_msg.append(applyVisitor(FieldVisitorToString(), column[row_idx]));
first = false; first = false;
} }
throw Exception{exception_message.str(), ErrorCodes::VIOLATED_CONSTRAINT}; throw Exception(ErrorCodes::VIOLATED_CONSTRAINT,
"Constraint {} for table {} is violated at row {}. Expression: ({}). Column values: {}",
backQuote(constraint_ptr->name), table_id.getNameForLogs(), rows_written + row_idx + 1,
serializeAST(*(constraint_ptr->expr), true), column_values_msg);
} }
} }
} }

10
src/DataStreams/IBlockInputStream.cpp
View File

@ -4,7 +4,8 @@
#include <Interpreters/ProcessList.h> #include <Interpreters/ProcessList.h>
#include <Access/EnabledQuota.h> #include <Access/EnabledQuota.h>
#include <Common/CurrentThread.h> #include <Common/CurrentThread.h>
#include <common/sleep.h> #include <IO/WriteBufferFromString.h>
#include <IO/Operators.h>
namespace ProfileEvents namespace ProfileEvents
{ {
@ -359,8 +360,7 @@ Block IBlockInputStream::getExtremes()
String IBlockInputStream::getTreeID() const String IBlockInputStream::getTreeID() const
{ {
std::stringstream s; WriteBufferFromOwnString s;
s.exceptions(std::ios::failbit);
s << getName(); s << getName();
if (!children.empty()) if (!children.empty())
@ -399,13 +399,13 @@ size_t IBlockInputStream::checkDepthImpl(size_t max_depth, size_t level) const
} }
void IBlockInputStream::dumpTree(std::ostream & ostr, size_t indent, size_t multiplier) const void IBlockInputStream::dumpTree(WriteBuffer & ostr, size_t indent, size_t multiplier) const
{ {
ostr << String(indent, ' ') << getName(); ostr << String(indent, ' ') << getName();
if (multiplier > 1) if (multiplier > 1)
ostr << " × " << multiplier; ostr << " × " << multiplier;
//ostr << ": " << getHeader().dumpStructure(); //ostr << ": " << getHeader().dumpStructure();
ostr << std::endl; ostr << '\n';
++indent; ++indent;
/// If the subtree is repeated several times, then we output it once with the multiplier. /// If the subtree is repeated several times, then we output it once with the multiplier.

2
src/DataStreams/IBlockInputStream.h
View File

@ -95,7 +95,7 @@ public:
virtual void readSuffix(); virtual void readSuffix();
/// Must be called before `read()` and `readPrefix()`. /// Must be called before `read()` and `readPrefix()`.
void dumpTree(std::ostream & ostr, size_t indent = 0, size_t multiplier = 1) const; void dumpTree(WriteBuffer & ostr, size_t indent = 0, size_t multiplier = 1) const;
/** Check the depth of the pipeline. /** Check the depth of the pipeline.
* If max_depth is specified and the `depth` is greater - throw an exception. * If max_depth is specified and the `depth` is greater - throw an exception.

2
src/DataStreams/MergingSortedBlockInputStream.cpp
View File

@ -1,6 +1,4 @@
#include <queue> #include <queue>
#include <iomanip>
#include <sstream>
#include <common/logger_useful.h> #include <common/logger_useful.h>

1
src/DataStreams/MongoDBBlockInputStream.cpp
View File

@ -1,4 +1,3 @@
#include <sstream>
#include <string> #include <string>
#include <vector> #include <vector>

3
src/DataTypes/DataTypeCustomSimpleAggregateFunction.cpp
View File

@ -32,8 +32,7 @@ static const std::vector<String> supported_functions{"any", "anyLast", "min",
String DataTypeCustomSimpleAggregateFunction::getName() const String DataTypeCustomSimpleAggregateFunction::getName() const
{ {
std::stringstream stream; WriteBufferFromOwnString stream;
stream.exceptions(std::ios::failbit);
stream << "SimpleAggregateFunction(" << function->getName(); stream << "SimpleAggregateFunction(" << function->getName();
if (!parameters.empty()) if (!parameters.empty())

5
src/DataTypes/DataTypesDecimal.cpp
View File

@ -29,10 +29,7 @@ namespace ErrorCodes
template <typename T> template <typename T>
std::string DataTypeDecimal<T>::doGetName() const std::string DataTypeDecimal<T>::doGetName() const
{ {
std::stringstream ss; return fmt::format("Decimal({}, {})", this->precision, this->scale);
ss.exceptions(std::ios::failbit);
ss << "Decimal(" << this->precision << ", " << this->scale << ")";
return ss.str();
} }

2
src/DataTypes/tests/gtest_data_type_get_common_type.cpp
View File

@ -26,7 +26,7 @@ static auto typeFromString(const std::string & str)
static auto typesFromString(const std::string & str) static auto typesFromString(const std::string & str)
{ {
std::istringstream data_types_stream(str); std::istringstream data_types_stream(str); // STYLE_CHECK_ALLOW_STD_STRING_STREAM
DataTypes data_types; DataTypes data_types;
std::string data_type; std::string data_type;
while (data_types_stream >> data_type) while (data_types_stream >> data_type)

16
src/Databases/DatabaseOnDisk.cpp
View File

@ -94,10 +94,9 @@ String getObjectDefinitionFromCreateQuery(const ASTPtr & query)
if (!create) if (!create)
{ {
std::ostringstream query_stream; WriteBufferFromOwnString query_buf;
query_stream.exceptions(std::ios::failbit); formatAST(*query, query_buf, true);
formatAST(*query, query_stream, true); throw Exception(ErrorCodes::LOGICAL_ERROR, "Query '{}' is not CREATE query", query_buf.str());
throw Exception("Query '" + query_stream.str() + "' is not CREATE query", ErrorCodes::LOGICAL_ERROR);
} }
if (!create->is_dictionary) if (!create->is_dictionary)
@ -121,11 +120,10 @@ String getObjectDefinitionFromCreateQuery(const ASTPtr & query)
if (create->uuid != UUIDHelpers::Nil) if (create->uuid != UUIDHelpers::Nil)
create->table = TABLE_WITH_UUID_NAME_PLACEHOLDER; create->table = TABLE_WITH_UUID_NAME_PLACEHOLDER;
std::ostringstream statement_stream; WriteBufferFromOwnString statement_buf;
statement_stream.exceptions(std::ios::failbit); formatAST(*create, statement_buf, false);
formatAST(*create, statement_stream, false); writeChar('\n', statement_buf);
statement_stream << '\n'; return statement_buf.str();
return statement_stream.str();
} }
DatabaseOnDisk::DatabaseOnDisk( DatabaseOnDisk::DatabaseOnDisk(

27
src/Databases/MySQL/MaterializeMySQLSyncThread.cpp
View File

@ -127,8 +127,7 @@ static String checkVariableAndGetVersion(const mysqlxx::Pool::Entry & connection
} }
bool first = true; bool first = true;
std::stringstream error_message; WriteBufferFromOwnString error_message;
error_message.exceptions(std::ios::failbit);
error_message << "Illegal MySQL variables, the MaterializeMySQL engine requires "; error_message << "Illegal MySQL variables, the MaterializeMySQL engine requires ";
for (const auto & [variable_name, variable_error_message] : variables_error_message) for (const auto & [variable_name, variable_error_message] : variables_error_message)
{ {
@ -239,8 +238,7 @@ static inline BlockOutputStreamPtr getTableOutput(const String & database_name,
{ {
const StoragePtr & storage = DatabaseCatalog::instance().getTable(StorageID(database_name, table_name), query_context); const StoragePtr & storage = DatabaseCatalog::instance().getTable(StorageID(database_name, table_name), query_context);
std::stringstream insert_columns_str; WriteBufferFromOwnString insert_columns_str;
insert_columns_str.exceptions(std::ios::failbit);
const StorageInMemoryMetadata & storage_metadata = storage->getInMemoryMetadata(); const StorageInMemoryMetadata & storage_metadata = storage->getInMemoryMetadata();
const ColumnsDescription & storage_columns = storage_metadata.getColumns(); const ColumnsDescription & storage_columns = storage_metadata.getColumns();
const NamesAndTypesList & insert_columns_names = insert_materialized ? storage_columns.getAllPhysical() : storage_columns.getOrdinary(); const NamesAndTypesList & insert_columns_names = insert_materialized ? storage_columns.getAllPhysical() : storage_columns.getOrdinary();
@ -331,10 +329,9 @@ std::optional<MaterializeMetadata> MaterializeMySQLSyncThread::prepareSynchroniz
const auto & position_message = [&]() const auto & position_message = [&]()
{ {
std::stringstream ss; WriteBufferFromOwnString buf;
ss.exceptions(std::ios::failbit); position.dump(buf);
position.dump(ss); return buf.str();
return ss.str();
}; };
LOG_INFO(log, "MySQL dump database position: \n {}", position_message()); LOG_INFO(log, "MySQL dump database position: \n {}", position_message());
} }
@ -374,10 +371,9 @@ void MaterializeMySQLSyncThread::flushBuffersData(Buffers & buffers, Materialize
const auto & position_message = [&]() const auto & position_message = [&]()
{ {
std::stringstream ss; WriteBufferFromOwnString buf;
ss.exceptions(std::ios::failbit); client.getPosition().dump(buf);
client.getPosition().dump(ss); return buf.str();
return ss.str();
}; };
LOG_INFO(log, "MySQL executed position: \n {}", position_message()); LOG_INFO(log, "MySQL executed position: \n {}", position_message());
} }
@ -632,10 +628,9 @@ void MaterializeMySQLSyncThread::onEvent(Buffers & buffers, const BinlogEventPtr
{ {
const auto & dump_event_message = [&]() const auto & dump_event_message = [&]()
{ {
std::stringstream ss; WriteBufferFromOwnString buf;
ss.exceptions(std::ios::failbit); receive_event->dump(buf);
receive_event->dump(ss); return buf.str();
return ss.str();
}; };
LOG_DEBUG(log, "Skip MySQL event: \n {}", dump_event_message()); LOG_DEBUG(log, "Skip MySQL event: \n {}", dump_event_message());

4
src/Dictionaries/DictionaryStructure.cpp
View File

@ -4,6 +4,7 @@
#include <DataTypes/DataTypeNullable.h> #include <DataTypes/DataTypeNullable.h>
#include <Formats/FormatSettings.h> #include <Formats/FormatSettings.h>
#include <IO/WriteHelpers.h> #include <IO/WriteHelpers.h>
#include <IO/Operators.h>
#include <Common/StringUtils/StringUtils.h> #include <Common/StringUtils/StringUtils.h>
#include <numeric> #include <numeric>
@ -230,8 +231,7 @@ std::string DictionaryStructure::getKeyDescription() const
if (id) if (id)
return "UInt64"; return "UInt64";
std::ostringstream out; WriteBufferFromOwnString out;
out.exceptions(std::ios::failbit);
out << '('; out << '(';

2
src/Dictionaries/tests/gtest_dictionary_configuration.cpp
View File

@ -18,7 +18,7 @@ static bool registered = false;
static std::string configurationToString(const DictionaryConfigurationPtr & config) static std::string configurationToString(const DictionaryConfigurationPtr & config)
{ {
const Poco::Util::XMLConfiguration * xml_config = dynamic_cast<const Poco::Util::XMLConfiguration *>(config.get()); const Poco::Util::XMLConfiguration * xml_config = dynamic_cast<const Poco::Util::XMLConfiguration *>(config.get());
std::ostringstream oss; std::ostringstream oss; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
oss.exceptions(std::ios::failbit); oss.exceptions(std::ios::failbit);
xml_config->save(oss); xml_config->save(oss);
return oss.str(); return oss.str();

1
src/Functions/abtesting.cpp
View File

@ -3,7 +3,6 @@
#if !defined(ARCADIA_BUILD) && USE_STATS #if !defined(ARCADIA_BUILD) && USE_STATS
#include <math.h> #include <math.h>
#include <sstream>
#include <DataTypes/DataTypeString.h> #include <DataTypes/DataTypeString.h>
#include <Columns/ColumnString.h> #include <Columns/ColumnString.h>

47
src/IO/CompressionMethod.cpp
View File

@ -1,11 +1,13 @@
#include <IO/CompressionMethod.h> #include <IO/CompressionMethod.h>
#include <IO/ReadBuffer.h>
#include <IO/WriteBuffer.h>
#include <IO/ZlibInflatingReadBuffer.h>
#include <IO/ZlibDeflatingWriteBuffer.h>
#include <IO/BrotliReadBuffer.h> #include <IO/BrotliReadBuffer.h>
#include <IO/BrotliWriteBuffer.h> #include <IO/BrotliWriteBuffer.h>
#include <IO/LZMADeflatingWriteBuffer.h>
#include <IO/LZMAInflatingReadBuffer.h>
#include <IO/ReadBuffer.h>
#include <IO/WriteBuffer.h>
#include <IO/ZlibDeflatingWriteBuffer.h>
#include <IO/ZlibInflatingReadBuffer.h>
#if !defined(ARCADIA_BUILD) #if !defined(ARCADIA_BUILD)
# include <Common/config.h> # include <Common/config.h>
@ -14,7 +16,6 @@
namespace DB namespace DB
{ {
namespace ErrorCodes namespace ErrorCodes
{ {
extern const int NOT_IMPLEMENTED; extern const int NOT_IMPLEMENTED;
@ -25,10 +26,16 @@ std::string toContentEncodingName(CompressionMethod method)
{ {
switch (method) switch (method)
{ {
case CompressionMethod::Gzip: return "gzip"; case CompressionMethod::Gzip:
case CompressionMethod::Zlib: return "deflate"; return "gzip";
case CompressionMethod::Brotli: return "br"; case CompressionMethod::Zlib:
case CompressionMethod::None: return ""; return "deflate";
case CompressionMethod::Brotli:
return "br";
case CompressionMethod::Xz:
return "xz";
case CompressionMethod::None:
return "";
} }
__builtin_unreachable(); __builtin_unreachable();
} }
@ -52,20 +59,19 @@ CompressionMethod chooseCompressionMethod(const std::string & path, const std::s
return CompressionMethod::Zlib; return CompressionMethod::Zlib;
if (*method_str == "brotli" || *method_str == "br") if (*method_str == "brotli" || *method_str == "br")
return CompressionMethod::Brotli; return CompressionMethod::Brotli;
if (*method_str == "LZMA" || *method_str == "xz")
return CompressionMethod::Xz;
if (hint.empty() || hint == "auto" || hint == "none") if (hint.empty() || hint == "auto" || hint == "none")
return CompressionMethod::None; return CompressionMethod::None;
throw Exception("Unknown compression method " + hint + ". Only 'auto', 'none', 'gzip', 'br' are supported as compression methods", throw Exception(
"Unknown compression method " + hint + ". Only 'auto', 'none', 'gzip', 'br', 'xz' are supported as compression methods",
ErrorCodes::NOT_IMPLEMENTED); ErrorCodes::NOT_IMPLEMENTED);
} }
std::unique_ptr<ReadBuffer> wrapReadBufferWithCompressionMethod( std::unique_ptr<ReadBuffer> wrapReadBufferWithCompressionMethod(
std::unique_ptr<ReadBuffer> nested, std::unique_ptr<ReadBuffer> nested, CompressionMethod method, size_t buf_size, char * existing_memory, size_t alignment)
CompressionMethod method,
size_t buf_size,
char * existing_memory,
size_t alignment)
{ {
if (method == CompressionMethod::Gzip || method == CompressionMethod::Zlib) if (method == CompressionMethod::Gzip || method == CompressionMethod::Zlib)
return std::make_unique<ZlibInflatingReadBuffer>(std::move(nested), method, buf_size, existing_memory, alignment); return std::make_unique<ZlibInflatingReadBuffer>(std::move(nested), method, buf_size, existing_memory, alignment);
@ -73,6 +79,8 @@ std::unique_ptr<ReadBuffer> wrapReadBufferWithCompressionMethod(
if (method == CompressionMethod::Brotli) if (method == CompressionMethod::Brotli)
return std::make_unique<BrotliReadBuffer>(std::move(nested), buf_size, existing_memory, alignment); return std::make_unique<BrotliReadBuffer>(std::move(nested), buf_size, existing_memory, alignment);
#endif #endif
if (method == CompressionMethod::Xz)
return std::make_unique<LZMAInflatingReadBuffer>(std::move(nested), buf_size, existing_memory, alignment);
if (method == CompressionMethod::None) if (method == CompressionMethod::None)
return nested; return nested;
@ -82,12 +90,7 @@ std::unique_ptr<ReadBuffer> wrapReadBufferWithCompressionMethod(
std::unique_ptr<WriteBuffer> wrapWriteBufferWithCompressionMethod( std::unique_ptr<WriteBuffer> wrapWriteBufferWithCompressionMethod(
std::unique_ptr<WriteBuffer> nested, std::unique_ptr<WriteBuffer> nested, CompressionMethod method, int level, size_t buf_size, char * existing_memory, size_t alignment)
CompressionMethod method,
int level,
size_t buf_size,
char * existing_memory,
size_t alignment)
{ {
if (method == DB::CompressionMethod::Gzip || method == CompressionMethod::Zlib) if (method == DB::CompressionMethod::Gzip || method == CompressionMethod::Zlib)
return std::make_unique<ZlibDeflatingWriteBuffer>(std::move(nested), method, level, buf_size, existing_memory, alignment); return std::make_unique<ZlibDeflatingWriteBuffer>(std::move(nested), method, level, buf_size, existing_memory, alignment);
@ -96,6 +99,8 @@ std::unique_ptr<WriteBuffer> wrapWriteBufferWithCompressionMethod(
if (method == DB::CompressionMethod::Brotli) if (method == DB::CompressionMethod::Brotli)
return std::make_unique<BrotliWriteBuffer>(std::move(nested), level, buf_size, existing_memory, alignment); return std::make_unique<BrotliWriteBuffer>(std::move(nested), level, buf_size, existing_memory, alignment);
#endif #endif
if (method == CompressionMethod::Xz)
return std::make_unique<LZMADeflatingWriteBuffer>(std::move(nested), level, buf_size, existing_memory, alignment);
if (method == CompressionMethod::None) if (method == CompressionMethod::None)
return nested; return nested;

6
src/IO/CompressionMethod.h
View File

@ -1,14 +1,13 @@
#pragma once #pragma once
#include <string>
#include <memory> #include <memory>
#include <string>
#include <Core/Defines.h> #include <Core/Defines.h>
namespace DB namespace DB
{ {
class ReadBuffer; class ReadBuffer;
class WriteBuffer; class WriteBuffer;
@ -26,6 +25,9 @@ enum class CompressionMethod
/// DEFLATE compression with zlib header and Adler32 checksum. /// DEFLATE compression with zlib header and Adler32 checksum.
/// This option corresponds to HTTP Content-Encoding: deflate. /// This option corresponds to HTTP Content-Encoding: deflate.
Zlib, Zlib,
/// LZMA2-based content compression
/// This option corresponds to HTTP Content-Encoding: xz
Xz,
Brotli Brotli
}; };

2
src/IO/HTTPCommon.cpp
View File

@ -239,7 +239,7 @@ void assertResponseIsOk(const Poco::Net::HTTPRequest & request, Poco::Net::HTTPR
if (!(status == Poco::Net::HTTPResponse::HTTP_OK || (isRedirect(status) && allow_redirects))) if (!(status == Poco::Net::HTTPResponse::HTTP_OK || (isRedirect(status) && allow_redirects)))
{ {
std::stringstream error_message; std::stringstream error_message; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
error_message.exceptions(std::ios::failbit); error_message.exceptions(std::ios::failbit);
error_message << "Received error from remote server " << request.getURI() << ". HTTP status code: " << status << " " error_message << "Received error from remote server " << request.getURI() << ". HTTP status code: " << status << " "
<< response.getReason() << ", body: " << istr.rdbuf(); << response.getReason() << ", body: " << istr.rdbuf();

125
src/IO/LZMADeflatingWriteBuffer.cpp Normal file
View File

@ -0,0 +1,125 @@
#include <IO/LZMADeflatingWriteBuffer.h>
namespace DB
{
namespace ErrorCodes
{
extern const int LZMA_STREAM_ENCODER_FAILED;
}
LZMADeflatingWriteBuffer::LZMADeflatingWriteBuffer(
std::unique_ptr<WriteBuffer> out_, int compression_level, size_t buf_size, char * existing_memory, size_t alignment)
: BufferWithOwnMemory<WriteBuffer>(buf_size, existing_memory, alignment), out(std::move(out_))
{
lstr = LZMA_STREAM_INIT;
lstr.allocator = nullptr;
lstr.next_in = nullptr;
lstr.avail_in = 0;
lstr.next_out = nullptr;
lstr.avail_out = 0;
// options for further compression
lzma_options_lzma opt_lzma2;
if (lzma_lzma_preset(&opt_lzma2, compression_level))
throw Exception(ErrorCodes::LZMA_STREAM_ENCODER_FAILED, "lzma preset failed: lzma version: {}", LZMA_VERSION_STRING);
// LZMA_FILTER_X86 -
// LZMA2 - codec for *.xz files compression; LZMA is not suitable for this purpose
// VLI - variable length integer (in *.xz most integers encoded as VLI)
// LZMA_VLI_UNKNOWN (UINT64_MAX) - VLI value to denote that the value is unknown
lzma_filter filters[] = {
{.id = LZMA_FILTER_X86, .options = nullptr},
{.id = LZMA_FILTER_LZMA2, .options = &opt_lzma2},
{.id = LZMA_VLI_UNKNOWN, .options = nullptr},
};
lzma_ret ret = lzma_stream_encoder(&lstr, filters, LZMA_CHECK_CRC64);
if (ret != LZMA_OK)
throw Exception(
ErrorCodes::LZMA_STREAM_ENCODER_FAILED,
"lzma stream encoder init failed: error code: {} lzma version: {}",
ret,
LZMA_VERSION_STRING);
}
LZMADeflatingWriteBuffer::~LZMADeflatingWriteBuffer()
{
try
{
finish();
lzma_end(&lstr);
}
catch (...)
{
tryLogCurrentException(__PRETTY_FUNCTION__);
}
}
void LZMADeflatingWriteBuffer::nextImpl()
{
if (!offset())
return;
lstr.next_in = reinterpret_cast<unsigned char *>(working_buffer.begin());
lstr.avail_in = offset();
lzma_action action = LZMA_RUN;
do
{
out->nextIfAtEnd();
lstr.next_out = reinterpret_cast<unsigned char *>(out->position());
lstr.avail_out = out->buffer().end() - out->position();
lzma_ret ret = lzma_code(&lstr, action);
out->position() = out->buffer().end() - lstr.avail_out;
if (ret == LZMA_STREAM_END)
return;
if (ret != LZMA_OK)
throw Exception(
ErrorCodes::LZMA_STREAM_ENCODER_FAILED,
"lzma stream encoding failed: error code: {}; lzma_version: {}",
ret,
LZMA_VERSION_STRING);
} while (lstr.avail_in > 0 || lstr.avail_out == 0);
}
void LZMADeflatingWriteBuffer::finish()
{
if (finished)
return;
next();
do
{
out->nextIfAtEnd();
lstr.next_out = reinterpret_cast<unsigned char *>(out->position());
lstr.avail_out = out->buffer().end() - out->position();
lzma_ret ret = lzma_code(&lstr, LZMA_FINISH);
out->position() = out->buffer().end() - lstr.avail_out;
if (ret == LZMA_STREAM_END)
{
finished = true;
return;
}
if (ret != LZMA_OK)
throw Exception(
ErrorCodes::LZMA_STREAM_ENCODER_FAILED,
"lzma stream encoding failed: error code: {}; lzma version: {}",
ret,
LZMA_VERSION_STRING);
} while (lstr.avail_out == 0);
}
}

32
src/IO/LZMADeflatingWriteBuffer.h Normal file
View File

@ -0,0 +1,32 @@
#pragma once
#include <IO/BufferWithOwnMemory.h>
#include <IO/WriteBuffer.h>
#include <lzma.h>
namespace DB
{
/// Performs compression using lzma library and writes compressed data to out_ WriteBuffer.
class LZMADeflatingWriteBuffer : public BufferWithOwnMemory<WriteBuffer>
{
public:
LZMADeflatingWriteBuffer(
std::unique_ptr<WriteBuffer> out_,
int compression_level,
size_t buf_size = DBMS_DEFAULT_BUFFER_SIZE,
char * existing_memory = nullptr,
size_t alignment = 0);
void finish();
~LZMADeflatingWriteBuffer() override;
private:
void nextImpl() override;
std::unique_ptr<WriteBuffer> out;
lzma_stream lstr;
bool finished = false;
};
}

89
src/IO/LZMAInflatingReadBuffer.cpp Normal file
View File

@ -0,0 +1,89 @@
#include <IO/LZMAInflatingReadBuffer.h>
namespace DB
{
namespace ErrorCodes
{
extern const int LZMA_STREAM_DECODER_FAILED;
}
LZMAInflatingReadBuffer::LZMAInflatingReadBuffer(std::unique_ptr<ReadBuffer> in_, size_t buf_size, char * existing_memory, size_t alignment)
: BufferWithOwnMemory<ReadBuffer>(buf_size, existing_memory, alignment), in(std::move(in_)), eof(false)
{
lstr = LZMA_STREAM_INIT;
lstr.allocator = nullptr;
lstr.next_in = nullptr;
lstr.avail_in = 0;
lstr.next_out = nullptr;
lstr.avail_out = 0;
// 500 mb
uint64_t memlimit = 500ULL << 20;
lzma_ret ret = lzma_stream_decoder(&lstr, memlimit, LZMA_CONCATENATED);
// lzma does not provide api for converting error code to string unlike zlib
if (ret != LZMA_OK)
throw Exception(
ErrorCodes::LZMA_STREAM_DECODER_FAILED,
"lzma_stream_decoder initialization failed: error code: {}; lzma version: {}",
ret,
LZMA_VERSION_STRING);
}
LZMAInflatingReadBuffer::~LZMAInflatingReadBuffer()
{
lzma_end(&lstr);
}
bool LZMAInflatingReadBuffer::nextImpl()
{
if (eof)
return false;
lzma_action action = LZMA_RUN;
if (!lstr.avail_in)
{
in->nextIfAtEnd();
lstr.next_in = reinterpret_cast<unsigned char *>(in->position());
lstr.avail_in = in->buffer().end() - in->position();
}
if (in->eof())
{
action = LZMA_FINISH;
}
lstr.next_out = reinterpret_cast<unsigned char *>(internal_buffer.begin());
lstr.avail_out = internal_buffer.size();
lzma_ret ret = lzma_code(&lstr, action);
in->position() = in->buffer().end() - lstr.avail_in;
working_buffer.resize(internal_buffer.size() - lstr.avail_out);
if (ret == LZMA_STREAM_END)
{
if (in->eof())
{
eof = true;
return working_buffer.size() != 0;
}
else
{
throw Exception(
ErrorCodes::LZMA_STREAM_DECODER_FAILED,
"lzma decoder finished, but input stream has not exceeded: error code: {}; lzma version: {}",
ret,
LZMA_VERSION_STRING);
}
}
if (ret != LZMA_OK)
throw Exception(
ErrorCodes::LZMA_STREAM_DECODER_FAILED,
"lzma_stream_decoder failed: error code: error codeL {}; lzma version: {}",
ret,
LZMA_VERSION_STRING);
return true;
}
}

33
src/IO/LZMAInflatingReadBuffer.h Normal file
View File

@ -0,0 +1,33 @@
#pragma once
#include <IO/BufferWithOwnMemory.h>
#include <IO/ReadBuffer.h>
#include <lzma.h>
namespace DB
{
namespace ErrorCodes
{
}
class LZMAInflatingReadBuffer : public BufferWithOwnMemory<ReadBuffer>
{
public:
LZMAInflatingReadBuffer(
std::unique_ptr<ReadBuffer> in_,
size_t buf_size = DBMS_DEFAULT_BUFFER_SIZE,
char * existing_memory = nullptr,
size_t alignment = 0);
~LZMAInflatingReadBuffer() override;
private:
bool nextImpl() override;
std::unique_ptr<ReadBuffer> in;
lzma_stream lstr;
bool eof;
};
}

1
src/IO/MySQLPacketPayloadReadBuffer.cpp
View File

@ -1,5 +1,4 @@
#include <IO/MySQLPacketPayloadReadBuffer.h> #include <IO/MySQLPacketPayloadReadBuffer.h>
#include <sstream>
namespace DB namespace DB
{ {

3
src/IO/Operators.h
View File

@ -45,7 +45,9 @@ struct BinaryManipReadBuffer : std::reference_wrapper<ReadBuffer> { usin
template <typename T> WriteBuffer & operator<< (WriteBuffer & buf, const T & x) { writeText(x, buf); return buf; } template <typename T> WriteBuffer & operator<< (WriteBuffer & buf, const T & x) { writeText(x, buf); return buf; }
/// If you do not use the manipulators, the string is displayed without an escape, as is. /// If you do not use the manipulators, the string is displayed without an escape, as is.
template <> inline WriteBuffer & operator<< (WriteBuffer & buf, const String & x) { writeString(x, buf); return buf; } template <> inline WriteBuffer & operator<< (WriteBuffer & buf, const String & x) { writeString(x, buf); return buf; }
template <> inline WriteBuffer & operator<< (WriteBuffer & buf, const std::string_view & x) { writeString(StringRef(x), buf); return buf; }
template <> inline WriteBuffer & operator<< (WriteBuffer & buf, const char & x) { writeChar(x, buf); return buf; } template <> inline WriteBuffer & operator<< (WriteBuffer & buf, const char & x) { writeChar(x, buf); return buf; }
template <> inline WriteBuffer & operator<< (WriteBuffer & buf, const pcg32_fast & x) { PcgSerializer::serializePcg32(x, buf); return buf; }
inline WriteBuffer & operator<< (WriteBuffer & buf, const char * x) { writeCString(x, buf); return buf; } inline WriteBuffer & operator<< (WriteBuffer & buf, const char * x) { writeCString(x, buf); return buf; }
@ -73,6 +75,7 @@ inline WriteBuffer & operator<< (WriteBuffer & buf, FlushManip) { buf.next(); re
template <typename T> ReadBuffer & operator>> (ReadBuffer & buf, T & x) { readText(x, buf); return buf; } template <typename T> ReadBuffer & operator>> (ReadBuffer & buf, T & x) { readText(x, buf); return buf; }
template <> inline ReadBuffer & operator>> (ReadBuffer & buf, String & x) { readString(x, buf); return buf; } template <> inline ReadBuffer & operator>> (ReadBuffer & buf, String & x) { readString(x, buf); return buf; }
template <> inline ReadBuffer & operator>> (ReadBuffer & buf, char & x) { readChar(x, buf); return buf; } template <> inline ReadBuffer & operator>> (ReadBuffer & buf, char & x) { readChar(x, buf); return buf; }
template <> inline ReadBuffer & operator>> (ReadBuffer & buf, pcg32_fast & x) { PcgDeserializer::deserializePcg32(x, buf); return buf; }
/// If you specify a string literal for reading, this will mean - make sure there is a sequence of bytes and skip it. /// If you specify a string literal for reading, this will mean - make sure there is a sequence of bytes and skip it.
inline ReadBuffer & operator>> (ReadBuffer & buf, const char * x) { assertString(x, buf); return buf; } inline ReadBuffer & operator>> (ReadBuffer & buf, const char * x) { assertString(x, buf); return buf; }

Some files were not shown because too many files have changed in this diff Show More