mirror of
https://github.com/ClickHouse/ClickHouse.git
synced 2024-11-22 07:31:57 +00:00
Merge branch 'correctly_send_close_request' into complete_zk_api
This commit is contained in:
alesapin
commit
4aad6a597c
3
.gitmodules
vendored
3
.gitmodules
vendored
@ -193,3 +193,6 @@
|
||||
[submodule "contrib/miniselect"]
|
||||
path = contrib/miniselect
|
||||
url = https://github.com/danlark1/miniselect
|
||||
[submodule "contrib/xz"]
|
||||
path = contrib/xz
|
||||
url = https://github.com/xz-mirror/xz
|
||||
|
||||
119
CHANGELOG.md
119
CHANGELOG.md
@ -1,3 +1,122 @@
|
||||
### ClickHouse release v20.11.2.1, 2020-11-11
|
||||
|
||||
#### Backward Incompatible Change
|
||||
|
||||
* If some `profile` was specified in `distributed_ddl` config section, then this profile could overwrite settings of `default` profile on server startup. It's fixed, now settings of distributed DDL queries should not affect global server settings. [#16635](https://github.com/ClickHouse/ClickHouse/pull/16635) ([tavplubix](https://github.com/tavplubix)).
|
||||
* Restrict to use of non-comparable data types (like `AggregateFunction`) in keys (Sorting key, Primary key, Partition key, and so on). [#16601](https://github.com/ClickHouse/ClickHouse/pull/16601) ([alesapin](https://github.com/alesapin)).
|
||||
* Remove `ANALYZE` and `AST` queries, and make the setting `enable_debug_queries` obsolete since now it is the part of full featured `EXPLAIN` query. [#16536](https://github.com/ClickHouse/ClickHouse/pull/16536) ([Ivan](https://github.com/abyss7)).
|
||||
* Aggregate functions `boundingRatio`, `rankCorr`, `retention`, `timeSeriesGroupSum`, `timeSeriesGroupRateSum`, `windowFunnel` were erroneously made case-insensitive. Now their names are made case sensitive as designed. Only functions that are specified in SQL standard or made for compatibility with other DBMS or functions similar to those should be case-insensitive. [#16407](https://github.com/ClickHouse/ClickHouse/pull/16407) ([alexey-milovidov](https://github.com/alexey-milovidov)).
|
||||
* Make `rankCorr` function return nan on insufficient data https://github.com/ClickHouse/ClickHouse/issues/16124. [#16135](https://github.com/ClickHouse/ClickHouse/pull/16135) ([hexiaoting](https://github.com/hexiaoting)).
|
||||
|
||||
#### New Feature
|
||||
|
||||
* Added support of LDAP as a user directory for locally non-existent users. [#12736](https://github.com/ClickHouse/ClickHouse/pull/12736) ([Denis Glazachev](https://github.com/traceon)).
|
||||
* Add `system.replicated_fetches` table which shows currently running background fetches. [#16428](https://github.com/ClickHouse/ClickHouse/pull/16428) ([alesapin](https://github.com/alesapin)).
|
||||
* Added setting `date_time_output_format`. [#15845](https://github.com/ClickHouse/ClickHouse/pull/15845) ([Maksim Kita](https://github.com/kitaisreal)).
|
||||
* Added minimal web UI to ClickHouse. [#16158](https://github.com/ClickHouse/ClickHouse/pull/16158) ([alexey-milovidov](https://github.com/alexey-milovidov)).
|
||||
* Allows to read/write Single protobuf message at once (w/o length-delimiters). [#15199](https://github.com/ClickHouse/ClickHouse/pull/15199) ([filimonov](https://github.com/filimonov)).
|
||||
* Added initial OpenTelemetry support. ClickHouse now accepts OpenTelemetry traceparent headers over Native and HTTP protocols, and passes them downstream in some cases. The trace spans for executed queries are saved into the `system.opentelemetry_span_log` table. [#14195](https://github.com/ClickHouse/ClickHouse/pull/14195) ([Alexander Kuzmenkov](https://github.com/akuzm)).
|
||||
* Allow specify primary key in column list of `CREATE TABLE` query. This is needed for compatibility with other SQL dialects. [#15823](https://github.com/ClickHouse/ClickHouse/pull/15823) ([Maksim Kita](https://github.com/kitaisreal)).
|
||||
* Implement `OFFSET offset_row_count {ROW | ROWS} FETCH {FIRST | NEXT} fetch_row_count {ROW | ROWS} {ONLY | WITH TIES}` in SELECT query with ORDER BY. This is the SQL-standard way to specify `LIMIT`. [#15855](https://github.com/ClickHouse/ClickHouse/pull/15855) ([hexiaoting](https://github.com/hexiaoting)).
|
||||
* `errorCodeToName` function - return variable name of the error (useful for analyzing query_log and similar). `system.errors` table - shows how many times errors has been happened (respects `system_events_show_zero_values`). [#16438](https://github.com/ClickHouse/ClickHouse/pull/16438) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Added function `untuple` which is a special function which can introduce new columns to the SELECT list by expanding a named tuple. [#16242](https://github.com/ClickHouse/ClickHouse/pull/16242) ([Nikolai Kochetov](https://github.com/KochetovNicolai), [Amos Bird](https://github.com/amosbird)).
|
||||
* Now we can provide identifiers via query parameters. And these parameters can be used as table objects or columns. [#16594](https://github.com/ClickHouse/ClickHouse/pull/16594) ([Amos Bird](https://github.com/amosbird)).
|
||||
* Added big integers (UInt256, Int128, Int256) and UUID data types support for MergeTree BloomFilter index. Big integers is an experimental feature. [#16642](https://github.com/ClickHouse/ClickHouse/pull/16642) ([Maksim Kita](https://github.com/kitaisreal)).
|
||||
* Add `farmFingerprint64` function (non-cryptographic string hashing). [#16570](https://github.com/ClickHouse/ClickHouse/pull/16570) ([Jacob Hayes](https://github.com/JacobHayes)).
|
||||
* Add `log_queries_min_query_duration_ms`, only queries slower then the value of this setting will go to `query_log`/`query_thread_log` (i.e. something like `slow_query_log` in mysql). [#16529](https://github.com/ClickHouse/ClickHouse/pull/16529) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Ability to create a docker image on the top of `Alpine`. Uses precompiled binary and glibc components from ubuntu 20.04. [#16479](https://github.com/ClickHouse/ClickHouse/pull/16479) ([filimonov](https://github.com/filimonov)).
|
||||
* Added `toUUIDOrNull`, `toUUIDOrZero` cast functions. [#16337](https://github.com/ClickHouse/ClickHouse/pull/16337) ([Maksim Kita](https://github.com/kitaisreal)).
|
||||
* Add `max_concurrent_queries_for_all_users` setting, see [#6636](https://github.com/ClickHouse/ClickHouse/issues/6636) for use cases. [#16154](https://github.com/ClickHouse/ClickHouse/pull/16154) ([nvartolomei](https://github.com/nvartolomei)).
|
||||
* Add a new option `print_query_id` to clickhouse-client. It helps generate arbitrary strings with the current query id generated by the client. Also print query id in clickhouse-client by default. [#15809](https://github.com/ClickHouse/ClickHouse/pull/15809) ([Amos Bird](https://github.com/amosbird)).
|
||||
* Add `tid` and `logTrace` functions. This closes [#9434](https://github.com/ClickHouse/ClickHouse/issues/9434). [#15803](https://github.com/ClickHouse/ClickHouse/pull/15803) ([flynn](https://github.com/ucasFL)).
|
||||
* Add function `formatReadableTimeDelta` that format time delta to human readable string ... [#15497](https://github.com/ClickHouse/ClickHouse/pull/15497) ([Filipe Caixeta](https://github.com/filipecaixeta)).
|
||||
* Added `disable_merges` option for volumes in multi-disk configuration. [#13956](https://github.com/ClickHouse/ClickHouse/pull/13956) ([Vladimir Chebotarev](https://github.com/excitoon)).
|
||||
|
||||
#### Experimental Feature
|
||||
|
||||
* New functions `encrypt`, `aes_encrypt_mysql`, `decrypt`, `aes_decrypt_mysql`. These functions are working slowly, so we consider it as an experimental feature. [#11844](https://github.com/ClickHouse/ClickHouse/pull/11844) ([Vasily Nemkov](https://github.com/Enmk)).
|
||||
|
||||
#### Bug Fix
|
||||
|
||||
* Mask password in data_path in the `system.distribution_queue`. [#16727](https://github.com/ClickHouse/ClickHouse/pull/16727) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Fix `IN` operator over several columns and tuples with enabled `transform_null_in` setting. Fixes [#15310](https://github.com/ClickHouse/ClickHouse/issues/15310). [#16722](https://github.com/ClickHouse/ClickHouse/pull/16722) ([Anton Popov](https://github.com/CurtizJ)).
|
||||
* The setting `max_parallel_replicas` worked incorrectly if the queried table has no sampling. This fixes [#5733](https://github.com/ClickHouse/ClickHouse/issues/5733). [#16675](https://github.com/ClickHouse/ClickHouse/pull/16675) ([alexey-milovidov](https://github.com/alexey-milovidov)).
|
||||
* Fix optimize_read_in_order/optimize_aggregation_in_order with max_threads > 0 and expression in ORDER BY. [#16637](https://github.com/ClickHouse/ClickHouse/pull/16637) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Calculation of `DEFAULT` expressions was involving possible name collisions (that was very unlikely to encounter). This fixes [#9359](https://github.com/ClickHouse/ClickHouse/issues/9359). [#16612](https://github.com/ClickHouse/ClickHouse/pull/16612) ([alexey-milovidov](https://github.com/alexey-milovidov)).
|
||||
* Fix `query_thread_log.query_duration_ms` unit. [#16563](https://github.com/ClickHouse/ClickHouse/pull/16563) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Fix a bug when using MySQL Master -> MySQL Slave -> ClickHouse MaterializeMySQL Engine. `MaterializeMySQL` is an experimental feature. [#16504](https://github.com/ClickHouse/ClickHouse/pull/16504) ([TCeason](https://github.com/TCeason)).
|
||||
* Specifically crafted argument of `round` function with `Decimal` was leading to integer division by zero. This fixes [#13338](https://github.com/ClickHouse/ClickHouse/issues/13338). [#16451](https://github.com/ClickHouse/ClickHouse/pull/16451) ([alexey-milovidov](https://github.com/alexey-milovidov)).
|
||||
* Fix DROP TABLE for Distributed (racy with INSERT). [#16409](https://github.com/ClickHouse/ClickHouse/pull/16409) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Fix processing of very large entries in replication queue. Very large entries may appear in ALTER queries if table structure is extremely large (near 1 MB). This fixes [#16307](https://github.com/ClickHouse/ClickHouse/issues/16307). [#16332](https://github.com/ClickHouse/ClickHouse/pull/16332) ([alexey-milovidov](https://github.com/alexey-milovidov)).
|
||||
* Fixed the inconsistent behaviour when a part of return data could be dropped because the set for its filtration wasn't created. [#16308](https://github.com/ClickHouse/ClickHouse/pull/16308) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
|
||||
* Fix dictGet in sharding_key (and similar places, i.e. when the function context is stored permanently). [#16205](https://github.com/ClickHouse/ClickHouse/pull/16205) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Fix the exception thrown in `clickhouse-local` when trying to execute `OPTIMIZE` command. Fixes [#16076](https://github.com/ClickHouse/ClickHouse/issues/16076). [#16192](https://github.com/ClickHouse/ClickHouse/pull/16192) ([filimonov](https://github.com/filimonov)).
|
||||
* Fixes [#15780](https://github.com/ClickHouse/ClickHouse/issues/15780) regression, e.g. `indexOf([1, 2, 3], toLowCardinality(1))` now is prohibited but it should not be. [#16038](https://github.com/ClickHouse/ClickHouse/pull/16038) ([Mike](https://github.com/myrrc)).
|
||||
* Fix bug with MySQL database. When MySQL server used as database engine is down some queries raise Exception, because they try to get tables from disabled server, while it's unnecessary. For example, query `SELECT ... FROM system.parts` should work only with MergeTree tables and don't touch MySQL database at all. [#16032](https://github.com/ClickHouse/ClickHouse/pull/16032) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Now exception will be thrown when `ALTER MODIFY COLUMN ... DEFAULT ...` has incompatible default with column type. Fixes [#15854](https://github.com/ClickHouse/ClickHouse/issues/15854). [#15858](https://github.com/ClickHouse/ClickHouse/pull/15858) ([alesapin](https://github.com/alesapin)).
|
||||
* Fixed IPv4CIDRToRange/IPv6CIDRToRange functions to accept const IP-column values. [#15856](https://github.com/ClickHouse/ClickHouse/pull/15856) ([vladimir-golovchenko](https://github.com/vladimir-golovchenko)).
|
||||
|
||||
#### Improvement
|
||||
|
||||
* Treat `INTERVAL '1 hour'` as equivalent to `INTERVAL 1 HOUR`, to be compatible with Postgres and similar. This fixes [#15637](https://github.com/ClickHouse/ClickHouse/issues/15637). [#15978](https://github.com/ClickHouse/ClickHouse/pull/15978) ([flynn](https://github.com/ucasFL)).
|
||||
* Enable parsing enum values by their numeric ids for CSV, TSV and JSON input formats. [#15685](https://github.com/ClickHouse/ClickHouse/pull/15685) ([vivarum](https://github.com/vivarum)).
|
||||
* Better read task scheduling for JBOD architecture and `MergeTree` storage. New setting `read_backoff_min_concurrency` which serves as the lower limit to the number of reading threads. [#16423](https://github.com/ClickHouse/ClickHouse/pull/16423) ([Amos Bird](https://github.com/amosbird)).
|
||||
* Add missing support for `LowCardinality` in `Avro` format. [#16521](https://github.com/ClickHouse/ClickHouse/pull/16521) ([Mike](https://github.com/myrrc)).
|
||||
* Workaround for use `S3` with nginx server as proxy. Nginx currenty does not accept urls with empty path like `http://domain.com?delete`, but vanilla aws-sdk-cpp produces this kind of urls. This commit uses patched aws-sdk-cpp version, which makes urls with "/" as path in this cases, like `http://domain.com/?delete`. [#16814](https://github.com/ClickHouse/ClickHouse/pull/16814) ([ianton-ru](https://github.com/ianton-ru)).
|
||||
* Better diagnostics on parse errors in input data. Provide row number on `Cannot read all data` errors. [#16644](https://github.com/ClickHouse/ClickHouse/pull/16644) ([alexey-milovidov](https://github.com/alexey-milovidov)).
|
||||
* Make the behaviour of `minMap` and `maxMap` more desireable. It will not skip zero values in the result. Fixes [#16087](https://github.com/ClickHouse/ClickHouse/issues/16087). [#16631](https://github.com/ClickHouse/ClickHouse/pull/16631) ([Ildus Kurbangaliev](https://github.com/ildus)).
|
||||
* Better update of ZooKeeper configuration in runtime. [#16630](https://github.com/ClickHouse/ClickHouse/pull/16630) ([sundyli](https://github.com/sundy-li)).
|
||||
* Apply SETTINGS clause as early as possible. It allows to modify more settings in the query. This closes [#3178](https://github.com/ClickHouse/ClickHouse/issues/3178). [#16619](https://github.com/ClickHouse/ClickHouse/pull/16619) ([alexey-milovidov](https://github.com/alexey-milovidov)).
|
||||
* Now `event_time_microseconds` field stores in Decimal64, not UInt64. [#16617](https://github.com/ClickHouse/ClickHouse/pull/16617) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
|
||||
* Now paratmeterized functions can be used in `APPLY` column transformer. [#16589](https://github.com/ClickHouse/ClickHouse/pull/16589) ([Amos Bird](https://github.com/amosbird)).
|
||||
* Improve scheduling of background task which removes data of dropped tables in `Atomic` databases. `Atomic` databases do not create broken symlink to table data directory if table actually has no data directory. [#16584](https://github.com/ClickHouse/ClickHouse/pull/16584) ([tavplubix](https://github.com/tavplubix)).
|
||||
* Subqueries in `WITH` section (CTE) can reference previous subqueries in `WITH` section by their name. [#16575](https://github.com/ClickHouse/ClickHouse/pull/16575) ([Amos Bird](https://github.com/amosbird)).
|
||||
* Add current_database into `system.query_thread_log`. [#16558](https://github.com/ClickHouse/ClickHouse/pull/16558) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Allow to fetch parts that are already committed or outdated in the current instance into the detached directory. It's useful when migrating tables from another cluster and having N to 1 shards mapping. It's also consistent with the current fetchPartition implementation. [#16538](https://github.com/ClickHouse/ClickHouse/pull/16538) ([Amos Bird](https://github.com/amosbird)).
|
||||
* Multiple improvements for `RabbitMQ`: Fixed bug for [#16263](https://github.com/ClickHouse/ClickHouse/issues/16263). Also minimized event loop lifetime. Added more efficient queues setup. [#16426](https://github.com/ClickHouse/ClickHouse/pull/16426) ([Kseniia Sumarokova](https://github.com/kssenii)).
|
||||
* Fix debug assertion in `quantileDeterministic` function. In previous version it may also transfer up to two times more data over the network. Although no bug existed. This fixes [#15683](https://github.com/ClickHouse/ClickHouse/issues/15683). [#16410](https://github.com/ClickHouse/ClickHouse/pull/16410) ([alexey-milovidov](https://github.com/alexey-milovidov)).
|
||||
* Add `TablesToDropQueueSize` metric. It's equal to number of dropped tables, that are waiting for background data removal. [#16364](https://github.com/ClickHouse/ClickHouse/pull/16364) ([tavplubix](https://github.com/tavplubix)).
|
||||
* Better diagnostics when client has dropped connection. In previous versions, `Attempt to read after EOF` and `Broken pipe` exceptions were logged in server. In new version, it's information message `Client has dropped the connection, cancel the query.`. [#16329](https://github.com/ClickHouse/ClickHouse/pull/16329) ([alexey-milovidov](https://github.com/alexey-milovidov)).
|
||||
* Add total_rows/total_bytes (from system.tables) support for Set/Join table engines. [#16306](https://github.com/ClickHouse/ClickHouse/pull/16306) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Now it's possible to specify `PRIMARY KEY` without `ORDER BY` for MergeTree table engines family. Closes [#15591](https://github.com/ClickHouse/ClickHouse/issues/15591). [#16284](https://github.com/ClickHouse/ClickHouse/pull/16284) ([alesapin](https://github.com/alesapin)).
|
||||
* If there is no tmp folder in the system (chroot, misconfigutation etc) `clickhouse-local` will create temporary subfolder in the current directory. [#16280](https://github.com/ClickHouse/ClickHouse/pull/16280) ([filimonov](https://github.com/filimonov)).
|
||||
* Add support for nested data types (like named tuple) as sub-types. Fixes [#15587](https://github.com/ClickHouse/ClickHouse/issues/15587). [#16262](https://github.com/ClickHouse/ClickHouse/pull/16262) ([Ivan](https://github.com/abyss7)).
|
||||
* Support for `database_atomic_wait_for_drop_and_detach_synchronously`/`NO DELAY`/`SYNC` for `DROP DATABASE`. [#16127](https://github.com/ClickHouse/ClickHouse/pull/16127) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Add `allow_nondeterministic_optimize_skip_unused_shards` (to allow non deterministic like `rand()` or `dictGet()` in sharding key). [#16105](https://github.com/ClickHouse/ClickHouse/pull/16105) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Fix `memory_profiler_step`/`max_untracked_memory` for queries via HTTP (test included). Fix the issue that adjusting this value globally in xml config does not help either, since those settings are not applied anyway, only default (4MB) value is [used](https://github.com/ClickHouse/ClickHouse/blob/17731245336d8c84f75e4c0894c5797ed7732190/src/Common/ThreadStatus.h#L104). Fix `query_id` for the most root ThreadStatus of the http query (by initializing QueryScope after reading query_id). [#16101](https://github.com/ClickHouse/ClickHouse/pull/16101) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Now it's allowed to execute `ALTER ... ON CLUSTER` queries regardless of the `<internal_replication>` setting in cluster config. [#16075](https://github.com/ClickHouse/ClickHouse/pull/16075) ([alesapin](https://github.com/alesapin)).
|
||||
* Fix rare issue when `clickhouse-client` may abort on exit due to loading of suggestions. This fixes [#16035](https://github.com/ClickHouse/ClickHouse/issues/16035). [#16047](https://github.com/ClickHouse/ClickHouse/pull/16047) ([alexey-milovidov](https://github.com/alexey-milovidov)).
|
||||
* Add support of `cache` layout for `Redis` dictionaries with complex key. [#15985](https://github.com/ClickHouse/ClickHouse/pull/15985) ([Anton Popov](https://github.com/CurtizJ)).
|
||||
* Fix query hang (endless loop) in case of misconfiguration (`connections_with_failover_max_tries` set to 0). [#15876](https://github.com/ClickHouse/ClickHouse/pull/15876) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Change level of some log messages from information to debug, so information messages will not appear for every query. This closes [#5293](https://github.com/ClickHouse/ClickHouse/issues/5293). [#15816](https://github.com/ClickHouse/ClickHouse/pull/15816) ([alexey-milovidov](https://github.com/alexey-milovidov)).
|
||||
* Remove `MemoryTrackingInBackground*` metrics to avoid potentially misleading results. This fixes [#15684](https://github.com/ClickHouse/ClickHouse/issues/15684). [#15813](https://github.com/ClickHouse/ClickHouse/pull/15813) ([alexey-milovidov](https://github.com/alexey-milovidov)).
|
||||
* Add reconnects to `zookeeper-dump-tree` tool. [#15711](https://github.com/ClickHouse/ClickHouse/pull/15711) ([alexey-milovidov](https://github.com/alexey-milovidov)).
|
||||
* Allow explicitly specify columns list in `CREATE TABLE table AS table_function(...)` query. Fixes [#9249](https://github.com/ClickHouse/ClickHouse/issues/9249) Fixes [#14214](https://github.com/ClickHouse/ClickHouse/issues/14214). [#14295](https://github.com/ClickHouse/ClickHouse/pull/14295) ([tavplubix](https://github.com/tavplubix)).
|
||||
|
||||
#### Performance Improvement
|
||||
|
||||
* Do not merge parts across partitions in SELECT FINAL. [#15938](https://github.com/ClickHouse/ClickHouse/pull/15938) ([Kruglov Pavel](https://github.com/Avogar)).
|
||||
* Improve performance of `-OrNull` and `-OrDefault` aggregate functions. [#16661](https://github.com/ClickHouse/ClickHouse/pull/16661) ([alexey-milovidov](https://github.com/alexey-milovidov)).
|
||||
* Improve performance of `quantileMerge`. In previous versions it was obnoxiously slow. This closes [#1463](https://github.com/ClickHouse/ClickHouse/issues/1463). [#16643](https://github.com/ClickHouse/ClickHouse/pull/16643) ([alexey-milovidov](https://github.com/alexey-milovidov)).
|
||||
* Improve performance of logical functions a little. [#16347](https://github.com/ClickHouse/ClickHouse/pull/16347) ([alexey-milovidov](https://github.com/alexey-milovidov)).
|
||||
* Improved performance of merges assignment in MergeTree table engines. Shouldn't be visible for the user. [#16191](https://github.com/ClickHouse/ClickHouse/pull/16191) ([alesapin](https://github.com/alesapin)).
|
||||
* Speedup hashed/sparse_hashed dictionary loading by preallocating the hash table. [#15454](https://github.com/ClickHouse/ClickHouse/pull/15454) ([Azat Khuzhin](https://github.com/azat)).
|
||||
* Now trivial count optimization becomes slightly non-trivial. Predicates that contain exact partition expr can be optimized too. This also fixes [#11092](https://github.com/ClickHouse/ClickHouse/issues/11092) which returns wrong count when `max_parallel_replicas > 1`. [#15074](https://github.com/ClickHouse/ClickHouse/pull/15074) ([Amos Bird](https://github.com/amosbird)).
|
||||
|
||||
#### Build/Testing/Packaging Improvement
|
||||
|
||||
* Add flaky check for stateless tests. It will detect potentially flaky functional tests in advance, before they are merged. [#16238](https://github.com/ClickHouse/ClickHouse/pull/16238) ([alesapin](https://github.com/alesapin)).
|
||||
* Use proper version for `croaring` instead of amalgamation. [#16285](https://github.com/ClickHouse/ClickHouse/pull/16285) ([sundyli](https://github.com/sundy-li)).
|
||||
* Improve generation of build files for `ya.make` build system (Arcadia). [#16700](https://github.com/ClickHouse/ClickHouse/pull/16700) ([alexey-milovidov](https://github.com/alexey-milovidov)).
|
||||
* Add MySQL BinLog file check tool for `MaterializeMySQL` database engine. `MaterializeMySQL` is an experimental feature. [#16223](https://github.com/ClickHouse/ClickHouse/pull/16223) ([Winter Zhang](https://github.com/zhang2014)).
|
||||
* Check for executable bit on non-executable files. People often accidentially commit executable files from Windows. [#15843](https://github.com/ClickHouse/ClickHouse/pull/15843) ([alexey-milovidov](https://github.com/alexey-milovidov)).
|
||||
* Check for `#pragma once` in headers. [#15818](https://github.com/ClickHouse/ClickHouse/pull/15818) ([alexey-milovidov](https://github.com/alexey-milovidov)).
|
||||
* Fix illegal code style `&vector[idx]` in libhdfs3. This fixes libcxx debug build. See also https://github.com/ClickHouse-Extras/libhdfs3/pull/8 . [#15815](https://github.com/ClickHouse/ClickHouse/pull/15815) ([Amos Bird](https://github.com/amosbird)).
|
||||
* Fix build of one miscellaneous example tool on Mac OS. Note that we don't build examples on Mac OS in our CI (we build only ClickHouse binary), so there is zero chance it will not break again. This fixes [#15804](https://github.com/ClickHouse/ClickHouse/issues/15804). [#15808](https://github.com/ClickHouse/ClickHouse/pull/15808) ([alexey-milovidov](https://github.com/alexey-milovidov)).
|
||||
* Simplify Sys/V init script. [#14135](https://github.com/ClickHouse/ClickHouse/pull/14135) ([alexey-milovidov](https://github.com/alexey-milovidov)).
|
||||
* Added `boost::program_options` to `db_generator` in order to increase its usability. This closes [#15940](https://github.com/ClickHouse/ClickHouse/issues/15940). [#15973](https://github.com/ClickHouse/ClickHouse/pull/15973) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)).
|
||||
|
||||
|
||||
## ClickHouse release 20.10
|
||||
|
||||
### ClickHouse release v20.10.3.30, 2020-10-28
|
||||
|
||||
@ -113,6 +113,12 @@
|
||||
|
||||
#include "pcg_extras.hpp"
|
||||
|
||||
namespace DB
|
||||
{
|
||||
struct PcgSerializer;
|
||||
struct PcgDeserializer;
|
||||
}
|
||||
|
||||
namespace pcg_detail {
|
||||
|
||||
using namespace pcg_extras;
|
||||
@ -557,6 +563,9 @@ public:
|
||||
engine<xtype1, itype1,
|
||||
output_mixin1, output_previous1,
|
||||
stream_mixin1, multiplier_mixin1>& rng);
|
||||
|
||||
friend ::DB::PcgSerializer;
|
||||
friend ::DB::PcgDeserializer;
|
||||
};
|
||||
|
||||
template <typename CharT, typename Traits,
|
||||
|
||||
1
contrib/CMakeLists.txt
vendored
1
contrib/CMakeLists.txt
vendored
@ -36,6 +36,7 @@ add_subdirectory (murmurhash)
|
||||
add_subdirectory (replxx-cmake)
|
||||
add_subdirectory (ryu-cmake)
|
||||
add_subdirectory (unixodbc-cmake)
|
||||
add_subdirectory (xz)
|
||||
|
||||
add_subdirectory (poco-cmake)
|
||||
add_subdirectory (croaring-cmake)
|
||||
|
||||
2
contrib/libunwind
vendored
2
contrib/libunwind
vendored
@ -1 +1 @@
|
||||
Subproject commit 27026ef4a9c6c8cc956d1d131c4d794e24096981
|
||||
Subproject commit 198458b35f100da32bd3e74c2a3ce8d236db299b
|
||||
1
contrib/xz
vendored
Submodule
1
contrib/xz
vendored
Submodule
@ -0,0 +1 @@
|
||||
Subproject commit 869b9d1b4edd6df07f819d360d306251f8147353
|
||||
@ -1,5 +1,5 @@
|
||||
# docker build -t yandex/clickhouse-fasttest .
|
||||
FROM ubuntu:19.10
|
||||
FROM ubuntu:20.04
|
||||
|
||||
ENV DEBIAN_FRONTEND=noninteractive LLVM_VERSION=10
|
||||
|
||||
|
||||
@ -127,7 +127,7 @@ function clone_submodules
|
||||
(
|
||||
cd "$FASTTEST_SOURCE"
|
||||
|
||||
SUBMODULES_TO_UPDATE=(contrib/boost contrib/zlib-ng contrib/libxml2 contrib/poco contrib/libunwind contrib/ryu contrib/fmtlib contrib/base64 contrib/cctz contrib/libcpuid contrib/double-conversion contrib/libcxx contrib/libcxxabi contrib/libc-headers contrib/lz4 contrib/zstd contrib/fastops contrib/rapidjson contrib/re2 contrib/sparsehash-c11 contrib/croaring contrib/miniselect)
|
||||
SUBMODULES_TO_UPDATE=(contrib/boost contrib/zlib-ng contrib/libxml2 contrib/poco contrib/libunwind contrib/ryu contrib/fmtlib contrib/base64 contrib/cctz contrib/libcpuid contrib/double-conversion contrib/libcxx contrib/libcxxabi contrib/libc-headers contrib/lz4 contrib/zstd contrib/fastops contrib/rapidjson contrib/re2 contrib/sparsehash-c11 contrib/croaring contrib/miniselect contrib/xz)
|
||||
|
||||
git submodule sync
|
||||
git submodule update --init --recursive "${SUBMODULES_TO_UPDATE[@]}"
|
||||
@ -268,6 +268,7 @@ TESTS_TO_SKIP=(
|
||||
protobuf
|
||||
secure
|
||||
sha256
|
||||
xz
|
||||
|
||||
# Not sure why these two fail even in sequential mode. Disabled for now
|
||||
# to make some progress.
|
||||
|
||||
@ -11,6 +11,7 @@ toc_title: Adopters
|
||||
| Company | Industry | Usecase | Cluster Size | (Un)Compressed Data Size<abbr title="of single replica"><sup>\*</sup></abbr> | Reference |
|
||||
|------------------------------------------------------------------------------------------------|---------------------------------|-----------------------|------------------------------------------------------------|------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| <a href="https://2gis.ru" class="favicon">2gis</a> | Maps | Monitoring | — | — | [Talk in Russian, July 2019](https://youtu.be/58sPkXfq6nw) |
|
||||
| <a href="https://getadmiral.com/" class="favicon">Admiral</a> | Martech | Engagement Management | — | — | [Webinar Slides, June 2020](https://altinity.com/presentations/2020/06/16/big-data-in-real-time-how-clickhouse-powers-admirals-visitor-relationships-for-publishers) |
|
||||
| <a href="https://cn.aliyun.com/" class="favicon">Alibaba Cloud</a> | Cloud | Managed Service | — | — | [Official Website](https://help.aliyun.com/product/144466.html) |
|
||||
| <a href="https://alohabrowser.com/" class="favicon">Aloha Browser</a> | Mobile App | Browser backend | — | — | [Slides in Russian, May 2019](https://presentations.clickhouse.tech/meetup22/aloha.pdf) |
|
||||
| <a href="https://amadeus.com/" class="favicon">Amadeus</a> | Travel | Analytics | — | — | [Press Release, April 2018](https://www.altinity.com/blog/2018/4/5/amadeus-technologies-launches-investment-and-insights-tool-based-on-machine-learning-and-strategy-algorithms) |
|
||||
@ -29,6 +30,7 @@ toc_title: Adopters
|
||||
| <a href="https://www.citadelsecurities.com/" class="favicon">Citadel Securities</a> | Finance | — | — | — | [Contribution, March 2019](https://github.com/ClickHouse/ClickHouse/pull/4774) |
|
||||
| <a href="https://city-mobil.ru" class="favicon">Citymobil</a> | Taxi | Analytics | — | — | [Blog Post in Russian, March 2020](https://habr.com/en/company/citymobil/blog/490660/) |
|
||||
| <a href="https://cloudflare.com" class="favicon">Cloudflare</a> | CDN | Traffic analysis | 36 servers | — | [Blog post, May 2017](https://blog.cloudflare.com/how-cloudflare-analyzes-1m-dns-queries-per-second/), [Blog post, March 2018](https://blog.cloudflare.com/http-analytics-for-6m-requests-per-second-using-clickhouse/) |
|
||||
| <a href="https://corporate.comcast.com/" class="favicon">Comcast</a> | Media | CDN Traffic Analysis | — | — | [ApacheCon 2019 Talk](https://www.youtube.com/watch?v=e9TZ6gFDjNg) |
|
||||
| <a href="https://contentsquare.com" class="favicon">ContentSquare</a> | Web analytics | Main product | — | — | [Blog post in French, November 2018](http://souslecapot.net/2018/11/21/patrick-chatain-vp-engineering-chez-contentsquare-penser-davantage-amelioration-continue-que-revolution-constante/) |
|
||||
| <a href="https://coru.net/" class="favicon">Corunet</a> | Analytics | Main product | — | — | [Slides in English, April 2019](https://github.com/ClickHouse/clickhouse-presentations/blob/master/meetup21/predictive_models.pdf) |
|
||||
| <a href="https://www.creditx.com" class="favicon">CraiditX 氪信</a> | Finance AI | Analysis | — | — | [Slides in English, November 2019](https://github.com/ClickHouse/clickhouse-presentations/blob/master/meetup33/udf.pptx) |
|
||||
@ -64,7 +66,8 @@ toc_title: Adopters
|
||||
| <a href="https://tech.mymarilyn.ru" class="favicon">Marilyn</a> | Advertising | Statistics | — | — | [Talk in Russian, June 2017](https://www.youtube.com/watch?v=iXlIgx2khwc) |
|
||||
| <a href="https://mellodesign.ru/" class="favicon">Mello</a> | Marketing | Analytics | 1 server | — | [Article, Oct 2020](https://vc.ru/marketing/166180-razrabotka-tipovogo-otcheta-skvoznoy-analitiki) |
|
||||
| <a href="https://www.messagebird.com" class="favicon">MessageBird</a> | Telecommunications | Statistics | — | — | [Slides in English, November 2018](https://github.com/ClickHouse/clickhouse-presentations/blob/master/meetup20/messagebird.pdf) |
|
||||
| <a href="https://www.mindsdb.com/" class="favicon">MindsDB</a> | Machine Learning | Main Product | — | — | [Official Website](https://www.mindsdb.com/blog/machine-learning-models-as-tables-in-ch) |
|
||||
| <a href="https://www.mindsdb.com/" class="favicon">MindsDB</a> | Machine Learning | Main Product | — | — | [Official Website](https://www.mindsdb.com/blog/machine-learning-models-as-tables-in-ch) |x
|
||||
| <a href="https://mux.com/" class="favicon">MUX</a> | Online Video | Video Analytics | — | — | [Talk in English, August 2019](https://altinity.com/presentations/2019/8/13/how-clickhouse-became-the-default-analytics-database-for-mux/) |
|
||||
| <a href="https://www.mgid.com/" class="favicon">MGID</a> | Ad network | Web-analytics | — | — | [Blog post in Russian, April 2020](http://gs-studio.com/news-about-it/32777----clickhouse---c) |
|
||||
| <a href="https://getnoc.com/" class="favicon">NOC Project</a> | Network Monitoring | Analytics | Main Product | — | [Official Website](https://getnoc.com/features/big-data/) |
|
||||
| <a href="https://www.nuna.com/" class="favicon">Nuna Inc.</a> | Health Data Analytics | — | — | — | [Talk in English, July 2020](https://youtu.be/GMiXCMFDMow?t=170) |
|
||||
|
||||
@ -46,6 +46,6 @@ toc_priority: 29
|
||||
|
||||
`Log` 引擎为表中的每一列使用不同的文件。`StripeLog` 将所有的数据存储在一个文件中。因此 `StripeLog` 引擎在操作系统中使用更少的描述符，但是 `Log` 引擎提供更高的读性能。
|
||||
|
||||
`TingLog` 引擎是该系列中最简单的引擎并且提供了最少的功能和最低的性能。`TingLog` 引擎不支持并行读取和并发数据访问，并将每一列存储在不同的文件中。它比其余两种支持并行读取的引擎的读取速度更慢，并且使用了和 `Log` 引擎同样多的描述符。你可以在简单的低负载的情景下使用它。
|
||||
`TinyLog` 引擎是该系列中最简单的引擎并且提供了最少的功能和最低的性能。`TinyLog` 引擎不支持并行读取和并发数据访问，并将每一列存储在不同的文件中。它比其余两种支持并行读取的引擎的读取速度更慢，并且使用了和 `Log` 引擎同样多的描述符。你可以在简单的低负载的情景下使用它。
|
||||
|
||||
[来源文章](https://clickhouse.tech/docs/en/operations/table_engines/log_family/) <!--hide-->
|
||||
|
||||
@ -1,5 +1,5 @@
|
||||
# 日志 {#log}
|
||||
# Log {#log}
|
||||
|
||||
日志与 TinyLog 的不同之处在于，«标记» 的小文件与列文件存在一起。这些标记写在每个数据块上，并且包含偏移量，这些偏移量指示从哪里开始读取文件以便跳过指定的行数。这使得可以在多个线程中读取表数据。对于并发数据访问，可以同时执行读取操作，而写入操作则阻塞读取和其它写入。Log 引擎不支持索引。同样，如果写入表失败，则该表将被破坏，并且从该表读取将返回错误。Log 引擎适用于临时数据，write-once 表以及测试或演示目的。
|
||||
`Log` 与 `TinyLog` 的不同之处在于，«标记» 的小文件与列文件存在一起。这些标记写在每个数据块上，并且包含偏移量，这些偏移量指示从哪里开始读取文件以便跳过指定的行数。这使得可以在多个线程中读取表数据。对于并发数据访问，可以同时执行读取操作，而写入操作则阻塞读取和其它写入。`Log`引擎不支持索引。同样，如果写入表失败，则该表将被破坏，并且从该表读取将返回错误。`Log`引擎适用于临时数据，write-once 表以及测试或演示目的。
|
||||
|
||||
[原始文章](https://clickhouse.tech/docs/zh/operations/table_engines/log/) <!--hide-->
|
||||
|
||||
@ -11,9 +11,9 @@
|
||||
|
||||
不要禁用超线程。 它有助于某些查询，但不适用于其他查询。
|
||||
|
||||
## 涡轮增压 {#turbo-boost}
|
||||
## 超频 {#turbo-boost}
|
||||
|
||||
强烈推荐涡轮增压。 它显着提高了典型负载的性能。
|
||||
强烈推荐超频(turbo-boost)。 它显着提高了典型负载的性能。
|
||||
您可以使用 `turbostat` 要查看负载下的CPU的实际时钟速率。
|
||||
|
||||
## CPU缩放调控器 {#cpu-scaling-governor}
|
||||
@ -39,18 +39,18 @@ echo 'performance' | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_gover
|
||||
|
||||
始终禁用交换文件。 不这样做的唯一原因是，如果您使用的ClickHouse在您的个人笔记本电脑。
|
||||
|
||||
## 巨大的页面 {#huge-pages}
|
||||
## 大页(Huge Pages) {#huge-pages}
|
||||
|
||||
始终禁用透明巨大的页面。 它会干扰内存分alloc，从而导致显着的性能下降。
|
||||
始终禁用透明大页(transparent huge pages)。 它会干扰内存分alloc，从而导致显着的性能下降。
|
||||
|
||||
``` bash
|
||||
echo 'never' | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
|
||||
```
|
||||
|
||||
使用 `perf top` 观看内核中用于内存管理的时间。
|
||||
永久巨大的页面也不需要被分配。
|
||||
使用 `perf top` 观察内核中用于内存管理的时间。
|
||||
永久大页(permanent huge pages)也不需要被分配。
|
||||
|
||||
## 存储子系统 {#storage-subsystem}
|
||||
## 存储系统 {#storage-subsystem}
|
||||
|
||||
如果您的预算允许您使用SSD，请使用SSD。
|
||||
如果没有，请使用硬盘。 SATA硬盘7200转就行了。
|
||||
@ -100,27 +100,27 @@ XFS也是合适的，但它还没有经过ClickHouse的彻底测试。
|
||||
|
||||
如果可能的话，至少使用一个10GB的网络。 1Gb也可以工作，但对于使用数十tb的数据修补副本或处理具有大量中间数据的分布式查询，情况会更糟。
|
||||
|
||||
## 动物园管理员 {#zookeeper}
|
||||
## Zookeeper {#zookeeper}
|
||||
|
||||
您可能已经将ZooKeeper用于其他目的。 您可以使用相同的zookeeper安装，如果它还没有超载。
|
||||
|
||||
It’s best to use a fresh version of ZooKeeper – 3.4.9 or later. The version in stable Linux distributions may be outdated.
|
||||
最好使用新版本的 Zookeeper – 3.4.9 或之后的版本. 稳定 Liunx 发行版中的 Zookeeper 版本可能是落后的。
|
||||
|
||||
You should never use manually written scripts to transfer data between different ZooKeeper clusters, because the result will be incorrect for sequential nodes. Never use the «zkcopy» utility for the same reason: https://github.com/ksprojects/zkcopy/issues/15
|
||||
你永远不该使用自己手写的脚本在不同的 Zookeeper 集群之间转移数据, 这可能会导致序列节点的数据不正确。出于同样的原因，永远不要使用 zkcopy 工具: https://github.com/ksprojects/zkcopy/issues/15
|
||||
|
||||
如果要将现有ZooKeeper集群分为两个，正确的方法是增加其副本的数量，然后将其重新配置为两个独立的集群。
|
||||
|
||||
不要在与ClickHouse相同的服务器上运行ZooKeeper。 由于ZooKeeper对延迟非常敏感，ClickHouse可能会利用所有可用的系统资源。
|
||||
不要在与ClickHouse相同的服务器上运行ZooKeeper。 因为ZooKeeper对延迟非常敏感，而ClickHouse可能会占用所有可用的系统资源。
|
||||
|
||||
使用默认设置，ZooKeeper是一个定时炸弹:
|
||||
默认设置下，ZooKeeper 就像是一个定时炸弹:
|
||||
|
||||
> 使用默认配置时，ZooKeeper服务器不会从旧快照和日志中删除文件（请参阅autopurge），这是操作员的责任。
|
||||
当使用默认配置时，ZooKeeper服务不会从旧快照和日志中删除文件（请参阅autopurge），这是操作员的责任。
|
||||
|
||||
必须拆除炸弹
|
||||
必须拆除炸弹。
|
||||
|
||||
下面的ZooKeeper（3.5.1）配置在Yandex中使用。梅地卡生产环境截至2017年5月20日:
|
||||
下面的ZooKeeper（3.5.1）配置在 Yandex.Metrica 的生产环境中使用截至2017年5月20日:
|
||||
|
||||
动物园cfg:
|
||||
zoo.cfg:
|
||||
|
||||
``` bash
|
||||
# http://hadoop.apache.org/zookeeper/docs/current/zookeeperAdmin.html
|
||||
@ -222,7 +222,7 @@ JAVA_OPTS="-Xms{{ '{{' }} cluster.get('xms','128M') {{ '}}' }} \
|
||||
-XX:+CMSParallelRemarkEnabled"
|
||||
```
|
||||
|
||||
盐初始化:
|
||||
Salt init:
|
||||
|
||||
description "zookeeper-{{ '{{' }} cluster['name'] {{ '}}' }} centralized coordination service"
|
||||
|
||||
|
||||
@ -54,6 +54,7 @@
|
||||
#include <IO/WriteHelpers.h>
|
||||
#include <IO/Operators.h>
|
||||
#include <IO/UseSSL.h>
|
||||
#include <IO/WriteBufferFromOStream.h>
|
||||
#include <DataStreams/AsynchronousBlockInputStream.h>
|
||||
#include <DataStreams/AddingDefaultsBlockInputStream.h>
|
||||
#include <DataStreams/InternalTextLogsRowOutputStream.h>
|
||||
@ -1158,13 +1159,13 @@ private:
|
||||
ASTPtr ast_to_process;
|
||||
try
|
||||
{
|
||||
std::stringstream dump_before_fuzz;
|
||||
WriteBufferFromOwnString dump_before_fuzz;
|
||||
fuzz_base->dumpTree(dump_before_fuzz);
|
||||
auto base_before_fuzz = fuzz_base->formatForErrorMessage();
|
||||
|
||||
ast_to_process = fuzz_base->clone();
|
||||
|
||||
std::stringstream dump_of_cloned_ast;
|
||||
WriteBufferFromOwnString dump_of_cloned_ast;
|
||||
ast_to_process->dumpTree(dump_of_cloned_ast);
|
||||
|
||||
// Run the original query as well.
|
||||
@ -1186,7 +1187,9 @@ private:
|
||||
fprintf(stderr, "dump of cloned ast:\n%s\n",
|
||||
dump_of_cloned_ast.str().c_str());
|
||||
fprintf(stderr, "dump after fuzz:\n");
|
||||
fuzz_base->dumpTree(std::cerr);
|
||||
WriteBufferFromOStream cerr_buf(std::cerr, 4096);
|
||||
fuzz_base->dumpTree(cerr_buf);
|
||||
cerr_buf.next();
|
||||
|
||||
fmt::print(stderr, "IAST::clone() is broken for some AST node. This is a bug. The original AST ('dump before fuzz') and its cloned copy ('dump of cloned AST') refer to the same nodes, which must never happen. This means that their parent node doesn't implement clone() correctly.");
|
||||
|
||||
@ -1529,7 +1532,9 @@ private:
|
||||
if (is_interactive)
|
||||
{
|
||||
std::cout << std::endl;
|
||||
formatAST(*res, std::cout);
|
||||
WriteBufferFromOStream res_buf(std::cout, 4096);
|
||||
formatAST(*res, res_buf);
|
||||
res_buf.next();
|
||||
std::cout << std::endl << std::endl;
|
||||
}
|
||||
|
||||
|
||||
@ -8,6 +8,7 @@
|
||||
#include <Core/Types.h>
|
||||
#include <IO/Operators.h>
|
||||
#include <IO/UseSSL.h>
|
||||
#include <IO/WriteBufferFromOStream.h>
|
||||
#include <Parsers/ASTExpressionList.h>
|
||||
#include <Parsers/ASTFunction.h>
|
||||
#include <Parsers/ASTIdentifier.h>
|
||||
@ -419,7 +420,9 @@ void QueryFuzzer::fuzzMain(ASTPtr & ast)
|
||||
fuzz(ast);
|
||||
|
||||
std::cout << std::endl;
|
||||
formatAST(*ast, std::cout, false /*highlight*/);
|
||||
WriteBufferFromOStream ast_buf(std::cout, 4096);
|
||||
formatAST(*ast, ast_buf, false /*highlight*/);
|
||||
ast_buf.next();
|
||||
std::cout << std::endl << std::endl;
|
||||
}
|
||||
|
||||
|
||||
@ -86,7 +86,7 @@ Suggest::Suggest()
|
||||
|
||||
void Suggest::loadImpl(Connection & connection, const ConnectionTimeouts & timeouts, size_t suggestion_limit)
|
||||
{
|
||||
std::stringstream query;
|
||||
std::stringstream query; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
|
||||
query << "SELECT DISTINCT arrayJoin(extractAll(name, '[\\\\w_]{2,}')) AS res FROM ("
|
||||
"SELECT name FROM system.functions"
|
||||
" UNION ALL "
|
||||
|
||||
@ -93,7 +93,7 @@ private:
|
||||
|
||||
void parse(const String & hint)
|
||||
{
|
||||
std::stringstream ss;
|
||||
std::stringstream ss; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
|
||||
ss << hint;
|
||||
String item;
|
||||
|
||||
|
||||
@ -162,7 +162,7 @@ void ClusterCopier::discoverShardPartitions(const ConnectionTimeouts & timeouts,
|
||||
|
||||
if (!missing_partitions.empty())
|
||||
{
|
||||
std::stringstream ss;
|
||||
WriteBufferFromOwnString ss;
|
||||
for (const String & missing_partition : missing_partitions)
|
||||
ss << " " << missing_partition;
|
||||
|
||||
|
||||
@ -13,7 +13,7 @@ using ConfigurationPtr = Poco::AutoPtr<Poco::Util::AbstractConfiguration>;
|
||||
|
||||
ConfigurationPtr getConfigurationFromXMLString(const std::string & xml_data)
|
||||
{
|
||||
std::stringstream ss(xml_data);
|
||||
std::stringstream ss(xml_data); // STYLE_CHECK_ALLOW_STD_STRING_STREAM
|
||||
Poco::XML::InputSource input_source{ss};
|
||||
return {new Poco::Util::XMLConfiguration{&input_source}};
|
||||
}
|
||||
|
||||
@ -394,12 +394,8 @@ inline ASTPtr TaskTable::rewriteReplicatedCreateQueryToPlain()
|
||||
|
||||
inline String DB::TaskShard::getDescription() const
|
||||
{
|
||||
std::stringstream ss;
|
||||
ss << "N" << numberInCluster()
|
||||
<< " (having a replica " << getHostNameExample()
|
||||
<< ", pull table " + getQuotedTable(task_table.table_pull)
|
||||
<< " of cluster " + task_table.cluster_pull_name << ")";
|
||||
return ss.str();
|
||||
return fmt::format("N{} (having a replica {}, pull table {} of cluster {}",
|
||||
numberInCluster(), getHostNameExample(), getQuotedTable(task_table.table_pull), task_table.cluster_pull_name);
|
||||
}
|
||||
|
||||
inline String DB::TaskShard::getHostNameExample() const
|
||||
|
||||
@ -6,6 +6,7 @@
|
||||
#include <IO/ReadBufferFromFileDescriptor.h>
|
||||
#include <IO/ReadHelpers.h>
|
||||
#include <IO/WriteBufferFromFileDescriptor.h>
|
||||
#include <IO/WriteBufferFromOStream.h>
|
||||
#include <Parsers/ParserQuery.h>
|
||||
#include <Parsers/parseQuery.h>
|
||||
#include <Parsers/formatAST.h>
|
||||
@ -129,7 +130,9 @@ int mainEntryClickHouseFormat(int argc, char ** argv)
|
||||
ASTPtr res = parseQueryAndMovePosition(parser, pos, end, "query", multiple, 0, DBMS_DEFAULT_MAX_PARSER_DEPTH);
|
||||
if (!quiet)
|
||||
{
|
||||
formatAST(*res, std::cout, hilite, oneline);
|
||||
WriteBufferFromOStream res_buf(std::cout, 4096);
|
||||
formatAST(*res, res_buf, hilite, oneline);
|
||||
res_buf.next();
|
||||
if (multiple)
|
||||
std::cout << "\n;\n";
|
||||
std::cout << std::endl;
|
||||
|
||||
@ -422,7 +422,7 @@ static const char * minimal_default_user_xml =
|
||||
|
||||
static ConfigurationPtr getConfigurationFromXMLString(const char * xml_data)
|
||||
{
|
||||
std::stringstream ss{std::string{xml_data}};
|
||||
std::stringstream ss{std::string{xml_data}}; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
|
||||
Poco::XML::InputSource input_source{ss};
|
||||
return {new Poco::Util::XMLConfiguration{&input_source}};
|
||||
}
|
||||
|
||||
@ -113,16 +113,16 @@ void ODBCColumnsInfoHandler::handleRequest(Poco::Net::HTTPServerRequest & reques
|
||||
|
||||
/// TODO Why not do SQLColumns instead?
|
||||
std::string name = schema_name.empty() ? backQuoteIfNeed(table_name) : backQuoteIfNeed(schema_name) + "." + backQuoteIfNeed(table_name);
|
||||
std::stringstream ss;
|
||||
WriteBufferFromOwnString buf;
|
||||
std::string input = "SELECT * FROM " + name + " WHERE 1 = 0";
|
||||
ParserQueryWithOutput parser;
|
||||
ASTPtr select = parseQuery(parser, input.data(), input.data() + input.size(), "", context_settings.max_query_size, context_settings.max_parser_depth);
|
||||
|
||||
IAST::FormatSettings settings(ss, true);
|
||||
IAST::FormatSettings settings(buf, true);
|
||||
settings.always_quote_identifiers = true;
|
||||
settings.identifier_quoting_style = getQuotingStyle(hdbc);
|
||||
select->format(settings);
|
||||
std::string query = ss.str();
|
||||
std::string query = buf.str();
|
||||
|
||||
LOG_TRACE(log, "Inferring structure with query '{}'", query);
|
||||
|
||||
|
||||
@ -32,12 +32,12 @@ namespace
|
||||
for (const auto & column : columns)
|
||||
query.columns->children.emplace_back(std::make_shared<ASTIdentifier>(column.name));
|
||||
|
||||
std::stringstream ss;
|
||||
IAST::FormatSettings settings(ss, true);
|
||||
WriteBufferFromOwnString buf;
|
||||
IAST::FormatSettings settings(buf, true);
|
||||
settings.always_quote_identifiers = true;
|
||||
settings.identifier_quoting_style = quoting;
|
||||
query.IAST::format(settings);
|
||||
return ss.str();
|
||||
return buf.str();
|
||||
}
|
||||
|
||||
std::string getQuestionMarks(size_t n)
|
||||
|
||||
@ -270,10 +270,10 @@ int Server::run()
|
||||
if (config().hasOption("help"))
|
||||
{
|
||||
Poco::Util::HelpFormatter help_formatter(Server::options());
|
||||
std::stringstream header;
|
||||
header << commandName() << " [OPTION] [-- [ARG]...]\n";
|
||||
header << "positional arguments can be used to rewrite config.xml properties, for example, --http_port=8010";
|
||||
help_formatter.setHeader(header.str());
|
||||
auto header_str = fmt::format("{} [OPTION] [-- [ARG]...]\n"
|
||||
"positional arguments can be used to rewrite config.xml properties, for example, --http_port=8010",
|
||||
commandName());
|
||||
help_formatter.setHeader(header_str);
|
||||
help_formatter.format(std::cout);
|
||||
return 0;
|
||||
}
|
||||
|
||||
@ -197,11 +197,13 @@ namespace
|
||||
boost::range::push_back(queries, InterpreterShowGrantsQuery::getAttachGrantQueries(entity));
|
||||
|
||||
/// Serialize the list of ATTACH queries to a string.
|
||||
std::stringstream ss;
|
||||
ss.exceptions(std::ios::failbit);
|
||||
WriteBufferFromOwnString buf;
|
||||
for (const ASTPtr & query : queries)
|
||||
ss << *query << ";\n";
|
||||
String file_contents = std::move(ss).str();
|
||||
{
|
||||
formatAST(*query, buf, false, true);
|
||||
buf.write(";\n", 2);
|
||||
}
|
||||
String file_contents = buf.str();
|
||||
|
||||
/// First we save *.tmp file and then we rename if everything's ok.
|
||||
auto tmp_file_path = std::filesystem::path{file_path}.replace_extension(".tmp");
|
||||
@ -353,7 +355,7 @@ String DiskAccessStorage::getStorageParamsJSON() const
|
||||
json.set("path", directory_path);
|
||||
if (readonly)
|
||||
json.set("readonly", readonly.load());
|
||||
std::ostringstream oss;
|
||||
std::ostringstream oss; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
|
||||
oss.exceptions(std::ios::failbit);
|
||||
Poco::JSON::Stringifier::stringify(json, oss);
|
||||
return oss.str();
|
||||
|
||||
@ -150,7 +150,7 @@ String LDAPAccessStorage::getStorageParamsJSON() const
|
||||
params_json.set("server", ldap_server);
|
||||
params_json.set("roles", default_role_names);
|
||||
|
||||
std::ostringstream oss;
|
||||
std::ostringstream oss; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
|
||||
oss.exceptions(std::ios::failbit);
|
||||
Poco::JSON::Stringifier::stringify(params_json, oss);
|
||||
|
||||
|
||||
@ -460,7 +460,7 @@ String UsersConfigAccessStorage::getStorageParamsJSON() const
|
||||
Poco::JSON::Object json;
|
||||
if (!path.empty())
|
||||
json.set("path", path);
|
||||
std::ostringstream oss;
|
||||
std::ostringstream oss; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
|
||||
oss.exceptions(std::ios::failbit);
|
||||
Poco::JSON::Stringifier::stringify(json, oss);
|
||||
return oss.str();
|
||||
|
||||
@ -2,6 +2,9 @@
|
||||
|
||||
#include <IO/ReadHelpers.h>
|
||||
#include <IO/WriteHelpers.h>
|
||||
#include <IO/ReadBufferFromString.h>
|
||||
#include <IO/WriteBufferFromString.h>
|
||||
#include <IO/Operators.h>
|
||||
|
||||
#include <DataTypes/DataTypeArray.h>
|
||||
#include <DataTypes/DataTypeString.h>
|
||||
@ -244,10 +247,9 @@ public:
|
||||
if constexpr (Trait::sampler == Sampler::RNG)
|
||||
{
|
||||
DB::writeIntBinary<size_t>(this->data(place).total_values, buf);
|
||||
std::ostringstream rng_stream;
|
||||
rng_stream.exceptions(std::ios::failbit);
|
||||
rng_stream << this->data(place).rng;
|
||||
DB::writeStringBinary(rng_stream.str(), buf);
|
||||
WriteBufferFromOwnString rng_buf;
|
||||
rng_buf << this->data(place).rng;
|
||||
DB::writeStringBinary(rng_buf.str(), buf);
|
||||
}
|
||||
|
||||
// TODO
|
||||
@ -275,9 +277,8 @@ public:
|
||||
DB::readIntBinary<size_t>(this->data(place).total_values, buf);
|
||||
std::string rng_string;
|
||||
DB::readStringBinary(rng_string, buf);
|
||||
std::istringstream rng_stream(rng_string);
|
||||
rng_stream.exceptions(std::ios::failbit);
|
||||
rng_stream >> this->data(place).rng;
|
||||
ReadBufferFromString rng_buf(rng_string);
|
||||
rng_buf >> this->data(place).rng;
|
||||
}
|
||||
|
||||
// TODO
|
||||
@ -565,10 +566,9 @@ public:
|
||||
if constexpr (Trait::sampler == Sampler::RNG)
|
||||
{
|
||||
DB::writeIntBinary<size_t>(data(place).total_values, buf);
|
||||
std::ostringstream rng_stream;
|
||||
rng_stream.exceptions(std::ios::failbit);
|
||||
rng_stream << data(place).rng;
|
||||
DB::writeStringBinary(rng_stream.str(), buf);
|
||||
WriteBufferFromOwnString rng_buf;
|
||||
rng_buf << data(place).rng;
|
||||
DB::writeStringBinary(rng_buf.str(), buf);
|
||||
}
|
||||
|
||||
// TODO
|
||||
@ -600,9 +600,8 @@ public:
|
||||
DB::readIntBinary<size_t>(data(place).total_values, buf);
|
||||
std::string rng_string;
|
||||
DB::readStringBinary(rng_string, buf);
|
||||
std::istringstream rng_stream(rng_string);
|
||||
rng_stream.exceptions(std::ios::failbit);
|
||||
rng_stream >> data(place).rng;
|
||||
ReadBufferFromString rng_buf(rng_string);
|
||||
rng_buf >> data(place).rng;
|
||||
}
|
||||
|
||||
// TODO
|
||||
|
||||
@ -1,7 +1,5 @@
|
||||
#pragma once
|
||||
|
||||
#include <iostream>
|
||||
#include <sstream>
|
||||
#include <unordered_set>
|
||||
#include <Columns/ColumnsNumber.h>
|
||||
#include <Columns/ColumnArray.h>
|
||||
|
||||
@ -1,10 +1,8 @@
|
||||
#pragma once
|
||||
|
||||
#include <bitset>
|
||||
#include <iostream>
|
||||
#include <map>
|
||||
#include <queue>
|
||||
#include <sstream>
|
||||
#include <unordered_set>
|
||||
#include <utility>
|
||||
#include <Columns/ColumnArray.h>
|
||||
|
||||
@ -1,7 +1,5 @@
|
||||
#pragma once
|
||||
|
||||
#include <iostream>
|
||||
#include <sstream>
|
||||
#include <unordered_set>
|
||||
#include <Columns/ColumnsNumber.h>
|
||||
#include <DataTypes/DataTypeDateTime.h>
|
||||
|
||||
@ -3,11 +3,13 @@
|
||||
#include <limits>
|
||||
#include <algorithm>
|
||||
#include <climits>
|
||||
#include <sstream>
|
||||
#include <common/types.h>
|
||||
#include <IO/ReadBuffer.h>
|
||||
#include <IO/ReadHelpers.h>
|
||||
#include <IO/WriteHelpers.h>
|
||||
#include <IO/ReadBufferFromString.h>
|
||||
#include <IO/WriteBufferFromString.h>
|
||||
#include <IO/Operators.h>
|
||||
#include <Common/PODArray.h>
|
||||
#include <Common/NaNUtils.h>
|
||||
#include <Poco/Exception.h>
|
||||
@ -190,9 +192,8 @@ public:
|
||||
|
||||
std::string rng_string;
|
||||
DB::readStringBinary(rng_string, buf);
|
||||
std::istringstream rng_stream(rng_string);
|
||||
rng_stream.exceptions(std::ios::failbit);
|
||||
rng_stream >> rng;
|
||||
DB::ReadBufferFromString rng_buf(rng_string);
|
||||
rng_buf >> rng;
|
||||
|
||||
for (size_t i = 0; i < samples.size(); ++i)
|
||||
DB::readBinary(samples[i], buf);
|
||||
@ -205,10 +206,9 @@ public:
|
||||
DB::writeIntBinary<size_t>(sample_count, buf);
|
||||
DB::writeIntBinary<size_t>(total_values, buf);
|
||||
|
||||
std::ostringstream rng_stream;
|
||||
rng_stream.exceptions(std::ios::failbit);
|
||||
rng_stream << rng;
|
||||
DB::writeStringBinary(rng_stream.str(), buf);
|
||||
DB::WriteBufferFromOwnString rng_buf;
|
||||
rng_buf << rng;
|
||||
DB::writeStringBinary(rng_buf.str(), buf);
|
||||
|
||||
for (size_t i = 0; i < std::min(sample_count, total_values); ++i)
|
||||
DB::writeBinary(samples[i], buf);
|
||||
|
||||
@ -3,7 +3,6 @@
|
||||
#include <limits>
|
||||
#include <algorithm>
|
||||
#include <climits>
|
||||
#include <sstream>
|
||||
#include <AggregateFunctions/ReservoirSampler.h>
|
||||
#include <common/types.h>
|
||||
#include <Common/HashTable/Hash.h>
|
||||
|
||||
@ -330,6 +330,13 @@ if (ZSTD_LIBRARY)
|
||||
endif ()
|
||||
endif()
|
||||
|
||||
set (LZMA_LIBRARY liblzma)
|
||||
set (LZMA_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/contrib/xz/src/liblzma/api)
|
||||
if (LZMA_LIBRARY)
|
||||
target_link_libraries (clickhouse_common_io PUBLIC ${LZMA_LIBRARY})
|
||||
target_include_directories (clickhouse_common_io SYSTEM BEFORE PUBLIC ${LZMA_INCLUDE_DIR})
|
||||
endif()
|
||||
|
||||
if (USE_ICU)
|
||||
dbms_target_link_libraries (PRIVATE ${ICU_LIBRARIES})
|
||||
dbms_target_include_directories (SYSTEM PRIVATE ${ICU_INCLUDE_DIRS})
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
#include <Client/MultiplexedConnections.h>
|
||||
#include <IO/ConnectionTimeouts.h>
|
||||
#include <IO/Operators.h>
|
||||
#include <Common/thread_local_rng.h>
|
||||
|
||||
|
||||
@ -222,19 +223,18 @@ std::string MultiplexedConnections::dumpAddresses() const
|
||||
std::string MultiplexedConnections::dumpAddressesUnlocked() const
|
||||
{
|
||||
bool is_first = true;
|
||||
std::ostringstream os;
|
||||
os.exceptions(std::ios::failbit);
|
||||
WriteBufferFromOwnString buf;
|
||||
for (const ReplicaState & state : replica_states)
|
||||
{
|
||||
const Connection * connection = state.connection;
|
||||
if (connection)
|
||||
{
|
||||
os << (is_first ? "" : "; ") << connection->getDescription();
|
||||
buf << (is_first ? "" : "; ") << connection->getDescription();
|
||||
is_first = false;
|
||||
}
|
||||
}
|
||||
|
||||
return os.str();
|
||||
return buf.str();
|
||||
}
|
||||
|
||||
Packet MultiplexedConnections::receivePacketUnlocked()
|
||||
|
||||
@ -71,7 +71,7 @@ void checkColumn(
|
||||
std::unordered_map<UInt32, T> map;
|
||||
size_t num_collisions = 0;
|
||||
|
||||
std::stringstream collisions_str;
|
||||
std::stringstream collisions_str; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
|
||||
collisions_str.exceptions(std::ios::failbit);
|
||||
|
||||
for (size_t i = 0; i < eq_class.size(); ++i)
|
||||
|
||||
@ -5,7 +5,6 @@
|
||||
#include <cstdlib>
|
||||
#include <cstring>
|
||||
#include <algorithm>
|
||||
#include <sstream>
|
||||
#include <functional>
|
||||
#include <filesystem>
|
||||
#include <Poco/DOM/Text.h>
|
||||
@ -17,6 +16,8 @@
|
||||
#include <Common/StringUtils/StringUtils.h>
|
||||
#include <Common/Exception.h>
|
||||
#include <common/getResource.h>
|
||||
#include <IO/WriteBufferFromString.h>
|
||||
#include <IO/Operators.h>
|
||||
|
||||
#define PREPROCESSED_SUFFIX "-preprocessed"
|
||||
|
||||
@ -537,8 +538,7 @@ XMLDocumentPtr ConfigProcessor::processConfig(
|
||||
if (has_zk_includes)
|
||||
*has_zk_includes = !contributing_zk_paths.empty();
|
||||
|
||||
std::stringstream comment;
|
||||
comment.exceptions(std::ios::failbit);
|
||||
WriteBufferFromOwnString comment;
|
||||
comment << " This file was generated automatically.\n";
|
||||
comment << " Do not edit it: it is likely to be discarded and generated again before it's read next time.\n";
|
||||
comment << " Files used to generate this file:";
|
||||
|
||||
@ -519,34 +519,35 @@
|
||||
M(550, CONDITIONAL_TREE_PARENT_NOT_FOUND) \
|
||||
M(551, ILLEGAL_PROJECTION_MANIPULATOR) \
|
||||
M(552, UNRECOGNIZED_ARGUMENTS) \
|
||||
\
|
||||
M(553, LZMA_STREAM_ENCODER_FAILED) \
|
||||
M(554, LZMA_STREAM_DECODER_FAILED) \
|
||||
\
|
||||
M(999, KEEPER_EXCEPTION) \
|
||||
M(1000, POCO_EXCEPTION) \
|
||||
M(1001, STD_EXCEPTION) \
|
||||
M(1002, UNKNOWN_EXCEPTION) \
|
||||
M(1002, UNKNOWN_EXCEPTION)
|
||||
|
||||
/* See END */
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
namespace ErrorCodes
|
||||
{
|
||||
#define M(VALUE, NAME) extern const Value NAME = VALUE;
|
||||
APPLY_FOR_ERROR_CODES(M)
|
||||
#undef M
|
||||
#define M(VALUE, NAME) extern const Value NAME = VALUE;
|
||||
APPLY_FOR_ERROR_CODES(M)
|
||||
#undef M
|
||||
|
||||
constexpr Value END = 3000;
|
||||
std::atomic<Value> values[END + 1] {};
|
||||
std::atomic<Value> values[END + 1]{};
|
||||
|
||||
struct ErrorCodesNames
|
||||
{
|
||||
std::string_view names[END + 1];
|
||||
ErrorCodesNames()
|
||||
{
|
||||
#define M(VALUE, NAME) names[VALUE] = std::string_view(#NAME);
|
||||
APPLY_FOR_ERROR_CODES(M)
|
||||
#undef M
|
||||
#define M(VALUE, NAME) names[VALUE] = std::string_view(#NAME);
|
||||
APPLY_FOR_ERROR_CODES(M)
|
||||
#undef M
|
||||
}
|
||||
} error_codes_names;
|
||||
|
||||
@ -557,7 +558,7 @@ namespace ErrorCodes
|
||||
return error_codes_names.names[error_code];
|
||||
}
|
||||
|
||||
ErrorCode end() { return END+1; }
|
||||
ErrorCode end() { return END + 1; }
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@ -245,8 +245,7 @@ static std::string getExtraExceptionInfo(const std::exception & e)
|
||||
|
||||
std::string getCurrentExceptionMessage(bool with_stacktrace, bool check_embedded_stacktrace /*= false*/, bool with_extra_info /*= true*/)
|
||||
{
|
||||
std::stringstream stream;
|
||||
stream.exceptions(std::ios::failbit);
|
||||
WriteBufferFromOwnString stream;
|
||||
|
||||
try
|
||||
{
|
||||
@ -365,8 +364,7 @@ void tryLogException(std::exception_ptr e, Poco::Logger * logger, const std::str
|
||||
|
||||
std::string getExceptionMessage(const Exception & e, bool with_stacktrace, bool check_embedded_stacktrace)
|
||||
{
|
||||
std::stringstream stream;
|
||||
stream.exceptions(std::ios::failbit);
|
||||
WriteBufferFromOwnString stream;
|
||||
|
||||
try
|
||||
{
|
||||
|
||||
@ -16,13 +16,13 @@ struct HTMLForm : public Poco::Net::HTMLForm
|
||||
HTMLForm(const Poco::Net::HTTPRequest & request)
|
||||
{
|
||||
Poco::URI uri(request.getURI());
|
||||
std::istringstream istr(uri.getRawQuery());
|
||||
std::istringstream istr(uri.getRawQuery()); // STYLE_CHECK_ALLOW_STD_STRING_STREAM
|
||||
readUrl(istr);
|
||||
}
|
||||
|
||||
HTMLForm(const Poco::URI & uri)
|
||||
{
|
||||
std::istringstream istr(uri.getRawQuery());
|
||||
std::istringstream istr(uri.getRawQuery()); // STYLE_CHECK_ALLOW_STD_STRING_STREAM
|
||||
readUrl(istr);
|
||||
}
|
||||
|
||||
|
||||
@ -133,17 +133,13 @@ void MemoryTracker::alloc(Int64 size)
|
||||
BlockerInThread untrack_lock;
|
||||
|
||||
ProfileEvents::increment(ProfileEvents::QueryMemoryLimitExceeded);
|
||||
std::stringstream message;
|
||||
message.exceptions(std::ios::failbit);
|
||||
message << "Memory tracker";
|
||||
if (const auto * description = description_ptr.load(std::memory_order_relaxed))
|
||||
message << " " << description;
|
||||
message << ": fault injected. Would use " << formatReadableSizeWithBinarySuffix(will_be)
|
||||
<< " (attempt to allocate chunk of " << size << " bytes)"
|
||||
<< ", maximum: " << formatReadableSizeWithBinarySuffix(current_hard_limit);
|
||||
|
||||
const auto * description = description_ptr.load(std::memory_order_relaxed);
|
||||
amount.fetch_sub(size, std::memory_order_relaxed);
|
||||
throw DB::Exception(message.str(), DB::ErrorCodes::MEMORY_LIMIT_EXCEEDED);
|
||||
throw DB::Exception(DB::ErrorCodes::MEMORY_LIMIT_EXCEEDED,
|
||||
"Memory tracker{}{}: fault injected. Would use {} (attempt to allocate chunk of {} bytes), maximum: {}",
|
||||
description ? " " : "", description ? description : "",
|
||||
formatReadableSizeWithBinarySuffix(will_be),
|
||||
size, formatReadableSizeWithBinarySuffix(current_hard_limit));
|
||||
}
|
||||
|
||||
if (unlikely(current_profiler_limit && will_be > current_profiler_limit))
|
||||
@ -166,17 +162,13 @@ void MemoryTracker::alloc(Int64 size)
|
||||
BlockerInThread untrack_lock;
|
||||
|
||||
ProfileEvents::increment(ProfileEvents::QueryMemoryLimitExceeded);
|
||||
std::stringstream message;
|
||||
message.exceptions(std::ios::failbit);
|
||||
message << "Memory limit";
|
||||
if (const auto * description = description_ptr.load(std::memory_order_relaxed))
|
||||
message << " " << description;
|
||||
message << " exceeded: would use " << formatReadableSizeWithBinarySuffix(will_be)
|
||||
<< " (attempt to allocate chunk of " << size << " bytes)"
|
||||
<< ", maximum: " << formatReadableSizeWithBinarySuffix(current_hard_limit);
|
||||
|
||||
const auto * description = description_ptr.load(std::memory_order_relaxed);
|
||||
amount.fetch_sub(size, std::memory_order_relaxed);
|
||||
throw DB::Exception(message.str(), DB::ErrorCodes::MEMORY_LIMIT_EXCEEDED);
|
||||
throw DB::Exception(DB::ErrorCodes::MEMORY_LIMIT_EXCEEDED,
|
||||
"Memory limit{}{} exceeded: would use {} (attempt to allocate chunk of {} bytes), maximum: {}",
|
||||
description ? " " : "", description ? description : "",
|
||||
formatReadableSizeWithBinarySuffix(will_be),
|
||||
size, formatReadableSizeWithBinarySuffix(current_hard_limit));
|
||||
}
|
||||
|
||||
updatePeak(will_be);
|
||||
|
||||
@ -8,6 +8,7 @@
|
||||
#include <common/logger_useful.h>
|
||||
#include <common/errnoToString.h>
|
||||
#include <IO/WriteHelpers.h>
|
||||
#include <IO/Operators.h>
|
||||
#include <unistd.h>
|
||||
#include <csignal>
|
||||
|
||||
@ -73,8 +74,7 @@ ShellCommand::~ShellCommand()
|
||||
|
||||
void ShellCommand::logCommand(const char * filename, char * const argv[])
|
||||
{
|
||||
std::stringstream args;
|
||||
args.exceptions(std::ios::failbit);
|
||||
WriteBufferFromOwnString args;
|
||||
for (int i = 0; argv != nullptr && argv[i] != nullptr; ++i)
|
||||
{
|
||||
if (i > 0)
|
||||
|
||||
@ -23,7 +23,7 @@
|
||||
|
||||
std::string signalToErrorMessage(int sig, const siginfo_t & info, const ucontext_t & context)
|
||||
{
|
||||
std::stringstream error;
|
||||
std::stringstream error; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
|
||||
error.exceptions(std::ios::failbit);
|
||||
switch (sig)
|
||||
{
|
||||
@ -319,7 +319,7 @@ static void toStringEveryLineImpl(
|
||||
const DB::SymbolIndex & symbol_index = DB::SymbolIndex::instance();
|
||||
std::unordered_map<std::string, DB::Dwarf> dwarfs;
|
||||
|
||||
std::stringstream out;
|
||||
std::stringstream out; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
|
||||
out.exceptions(std::ios::failbit);
|
||||
|
||||
for (size_t i = offset; i < size; ++i)
|
||||
@ -359,7 +359,7 @@ static void toStringEveryLineImpl(
|
||||
out.str({});
|
||||
}
|
||||
#else
|
||||
std::stringstream out;
|
||||
std::stringstream out; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
|
||||
out.exceptions(std::ios::failbit);
|
||||
|
||||
for (size_t i = offset; i < size; ++i)
|
||||
@ -375,7 +375,7 @@ static void toStringEveryLineImpl(
|
||||
|
||||
static std::string toStringImpl(const StackTrace::FramePointers & frame_pointers, size_t offset, size_t size)
|
||||
{
|
||||
std::stringstream out;
|
||||
std::stringstream out; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
|
||||
out.exceptions(std::ios::failbit);
|
||||
toStringEveryLineImpl(frame_pointers, offset, size, [&](const std::string & str) { out << str << '\n'; });
|
||||
return out.str();
|
||||
|
||||
@ -153,7 +153,7 @@ std::pair<bool, std::string> StudentTTest::compareAndReport(size_t confidence_le
|
||||
|
||||
double mean_confidence_interval = table_value * t_statistic;
|
||||
|
||||
std::stringstream ss;
|
||||
std::stringstream ss; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
|
||||
ss.exceptions(std::ios::failbit);
|
||||
|
||||
if (mean_difference > mean_confidence_interval && (mean_difference - mean_confidence_interval > 0.0001)) /// difference must be more than 0.0001, to take into account connection latency.
|
||||
|
||||
@ -413,7 +413,8 @@ std::vector<size_t> PerfEventsCounters::eventIndicesFromString(const std::string
|
||||
return result;
|
||||
}
|
||||
|
||||
std::istringstream iss(events_list);
|
||||
|
||||
std::istringstream iss(events_list); // STYLE_CHECK_ALLOW_STD_STRING_STREAM
|
||||
std::string event_name;
|
||||
while (std::getline(iss, event_name, ','))
|
||||
{
|
||||
|
||||
@ -1,5 +1,3 @@
|
||||
#include <sstream>
|
||||
|
||||
#include <Common/Exception.h>
|
||||
#include <Common/ThreadProfileEvents.h>
|
||||
#include <Common/QueryProfiler.h>
|
||||
@ -79,12 +77,10 @@ void ThreadStatus::assertState(const std::initializer_list<int> & permitted_stat
|
||||
return;
|
||||
}
|
||||
|
||||
std::stringstream ss;
|
||||
ss.exceptions(std::ios::failbit);
|
||||
ss << "Unexpected thread state " << getCurrentState();
|
||||
if (description)
|
||||
ss << ": " << description;
|
||||
throw Exception(ss.str(), ErrorCodes::LOGICAL_ERROR);
|
||||
throw Exception(ErrorCodes::LOGICAL_ERROR, "Unexpected thread state {}: {}", getCurrentState(), description);
|
||||
else
|
||||
throw Exception(ErrorCodes::LOGICAL_ERROR, "Unexpected thread state {}", getCurrentState());
|
||||
}
|
||||
|
||||
void ThreadStatus::attachInternalTextLogsQueue(const InternalTextLogsQueuePtr & logs_queue,
|
||||
|
||||
@ -1,11 +1,11 @@
|
||||
#pragma once
|
||||
|
||||
#include <tuple>
|
||||
#include <sstream>
|
||||
#include <iomanip>
|
||||
#include <city.h>
|
||||
|
||||
#include <Core/Types.h>
|
||||
#include <Common/hex.h>
|
||||
|
||||
#ifdef __SSE4_2__
|
||||
#include <nmmintrin.h>
|
||||
@ -48,10 +48,9 @@ struct UInt128
|
||||
|
||||
String toHexString() const
|
||||
{
|
||||
std::ostringstream os;
|
||||
os.exceptions(std::ios::failbit);
|
||||
os << std::setw(16) << std::setfill('0') << std::hex << high << low;
|
||||
return String(os.str());
|
||||
String res(2 * sizeof(UInt128), 0);
|
||||
writeHexUIntLowercase(*this, res.data());
|
||||
return res;
|
||||
}
|
||||
|
||||
bool inline operator== (const UInt128 rhs) const { return tuple() == rhs.tuple(); }
|
||||
|
||||
@ -1,6 +1,5 @@
|
||||
#pragma once
|
||||
|
||||
#include <sstream>
|
||||
#include <IO/ReadHelpers.h>
|
||||
#include <IO/ReadWriteBufferFromHTTP.h>
|
||||
#include <Interpreters/Context.h>
|
||||
@ -307,9 +306,6 @@ struct ODBCBridgeMixin
|
||||
std::vector<std::string> cmd_args;
|
||||
path.setFileName("clickhouse-odbc-bridge");
|
||||
|
||||
std::stringstream command;
|
||||
command.exceptions(std::ios::failbit);
|
||||
|
||||
#if !CLICKHOUSE_SPLIT_BINARY
|
||||
cmd_args.push_back("odbc-bridge");
|
||||
#endif
|
||||
|
||||
@ -215,7 +215,7 @@ std::pair<ResponsePtr, Undo> TestKeeperCreateRequest::process(TestKeeper::Contai
|
||||
auto seq_num = it->second.seq_num;
|
||||
++it->second.seq_num;
|
||||
|
||||
std::stringstream seq_num_str;
|
||||
std::stringstream seq_num_str; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
|
||||
seq_num_str.exceptions(std::ios::failbit);
|
||||
seq_num_str << std::setw(10) << std::setfill('0') << seq_num;
|
||||
|
||||
|
||||
@ -533,6 +533,7 @@ void ZooKeeper::sendThread()
|
||||
setThreadName("ZooKeeperSend");
|
||||
|
||||
auto prev_heartbeat_time = clock::now();
|
||||
bool tried_to_send_close = false;
|
||||
|
||||
try
|
||||
{
|
||||
@ -562,6 +563,15 @@ void ZooKeeper::sendThread()
|
||||
std::lock_guard lock(operations_mutex);
|
||||
operations[info.request->xid] = info;
|
||||
}
|
||||
else
|
||||
{
|
||||
/// We set this variable only once. If we will
|
||||
/// successfully send close, than this thread will just
|
||||
/// finish. If we will got an exception while sending
|
||||
/// close, than thread will also finish and finalization
|
||||
/// will be completed by some other thread.
|
||||
tried_to_send_close = true;
|
||||
}
|
||||
|
||||
if (info.watch)
|
||||
{
|
||||
@ -600,7 +610,13 @@ void ZooKeeper::sendThread()
|
||||
catch (...)
|
||||
{
|
||||
tryLogCurrentException(__PRETTY_FUNCTION__);
|
||||
finalize(true, false);
|
||||
/// If we have tried to send close and got an exception than
|
||||
/// finalization is already started by receiveThread and we cannot do
|
||||
/// anything better than just exit.
|
||||
///
|
||||
/// Otherwise we should correctly finalize
|
||||
if (!tried_to_send_close)
|
||||
finalize(true, false);
|
||||
}
|
||||
}
|
||||
|
||||
@ -814,7 +830,15 @@ void ZooKeeper::finalize(bool error_send, bool error_receive)
|
||||
if (expired)
|
||||
return;
|
||||
|
||||
active_session_metric_increment.destroy();
|
||||
auto expire_session = [&]
|
||||
{
|
||||
std::lock_guard lock(push_request_mutex);
|
||||
if (!expired)
|
||||
{
|
||||
expired = true;
|
||||
active_session_metric_increment.destroy();
|
||||
}
|
||||
};
|
||||
|
||||
try
|
||||
{
|
||||
@ -829,8 +853,7 @@ void ZooKeeper::finalize(bool error_send, bool error_receive)
|
||||
{
|
||||
/// This happens for example, when "Cannot push request to queue within operation timeout".
|
||||
/// Just mark session expired in case of error on close request.
|
||||
std::lock_guard lock(push_request_mutex);
|
||||
expired = true;
|
||||
expire_session();
|
||||
tryLogCurrentException(__PRETTY_FUNCTION__);
|
||||
}
|
||||
|
||||
@ -838,10 +861,8 @@ void ZooKeeper::finalize(bool error_send, bool error_receive)
|
||||
send_thread.join();
|
||||
}
|
||||
|
||||
{
|
||||
std::lock_guard lock(push_request_mutex);
|
||||
expired = true;
|
||||
}
|
||||
/// Set expired flag after we sent close event
|
||||
expire_session();
|
||||
|
||||
try
|
||||
{
|
||||
|
||||
@ -3,7 +3,6 @@
|
||||
#include <ext/scope_guard.h>
|
||||
#include <pthread.h>
|
||||
#include <cstdint>
|
||||
#include <sstream>
|
||||
|
||||
#if defined(__FreeBSD__)
|
||||
# include <pthread_np.h>
|
||||
@ -80,13 +79,8 @@ __attribute__((__weak__)) void checkStackSize()
|
||||
/// It's safe to assume that overflow in multiplying by two cannot occur.
|
||||
if (stack_size * 2 > max_stack_size)
|
||||
{
|
||||
std::stringstream message;
|
||||
message.exceptions(std::ios::failbit);
|
||||
message << "Stack size too large"
|
||||
<< ". Stack address: " << stack_address
|
||||
<< ", frame address: " << frame_address
|
||||
<< ", stack size: " << stack_size
|
||||
<< ", maximum stack size: " << max_stack_size;
|
||||
throw Exception(message.str(), ErrorCodes::TOO_DEEP_RECURSION);
|
||||
throw Exception(ErrorCodes::TOO_DEEP_RECURSION,
|
||||
"Stack size too large. Stack address: {}, frame address: {}, stack size: {}, maximum stack size: {}",
|
||||
stack_address, frame_address, stack_size, max_stack_size);
|
||||
}
|
||||
}
|
||||
|
||||
@ -1,6 +1,4 @@
|
||||
#include <cmath>
|
||||
#include <sstream>
|
||||
#include <iomanip>
|
||||
|
||||
#include <Common/formatReadable.h>
|
||||
#include <IO/DoubleConverter.h>
|
||||
|
||||
@ -1,4 +1,7 @@
|
||||
#include <Common/parseGlobs.h>
|
||||
#include <IO/WriteBufferFromString.h>
|
||||
#include <IO/ReadBufferFromString.h>
|
||||
#include <IO/Operators.h>
|
||||
#include <re2/re2.h>
|
||||
#include <re2/stringpiece.h>
|
||||
#include <algorithm>
|
||||
@ -18,21 +21,21 @@ namespace DB
|
||||
*/
|
||||
std::string makeRegexpPatternFromGlobs(const std::string & initial_str_with_globs)
|
||||
{
|
||||
std::ostringstream oss_for_escaping;
|
||||
oss_for_escaping.exceptions(std::ios::failbit);
|
||||
/// FIXME make it better
|
||||
WriteBufferFromOwnString buf_for_escaping;
|
||||
/// Escaping only characters that not used in glob syntax
|
||||
for (const auto & letter : initial_str_with_globs)
|
||||
{
|
||||
if ((letter == '[') || (letter == ']') || (letter == '|') || (letter == '+') || (letter == '-') || (letter == '(') || (letter == ')'))
|
||||
oss_for_escaping << '\\';
|
||||
oss_for_escaping << letter;
|
||||
if ((letter == '[') || (letter == ']') || (letter == '|') || (letter == '+') || (letter == '-') || (letter == '(') || (letter == ')') || (letter == '\\'))
|
||||
buf_for_escaping << '\\';
|
||||
buf_for_escaping << letter;
|
||||
}
|
||||
std::string escaped_with_globs = oss_for_escaping.str();
|
||||
std::string escaped_with_globs = buf_for_escaping.str();
|
||||
|
||||
static const re2::RE2 enum_or_range(R"({([\d]+\.\.[\d]+|[^{}*,]+,[^{}*]*[^{}*,])})"); /// regexp for {expr1,expr2,expr3} or {M..N}, where M and N - non-negative integers, expr's should be without {}*,
|
||||
re2::StringPiece input(escaped_with_globs);
|
||||
re2::StringPiece matched;
|
||||
std::ostringstream oss_for_replacing;
|
||||
std::ostringstream oss_for_replacing; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
|
||||
oss_for_replacing.exceptions(std::ios::failbit);
|
||||
size_t current_index = 0;
|
||||
while (RE2::FindAndConsume(&input, enum_or_range, &matched))
|
||||
@ -45,9 +48,8 @@ std::string makeRegexpPatternFromGlobs(const std::string & initial_str_with_glob
|
||||
size_t range_begin = 0;
|
||||
size_t range_end = 0;
|
||||
char point;
|
||||
std::istringstream iss_range(buffer);
|
||||
iss_range.exceptions(std::ios::failbit);
|
||||
iss_range >> range_begin >> point >> point >> range_end;
|
||||
ReadBufferFromString buf_range(buffer);
|
||||
buf_range >> range_begin >> point >> point >> range_end;
|
||||
bool leading_zeros = buffer[0] == '0';
|
||||
size_t num_len = std::to_string(range_end).size();
|
||||
if (leading_zeros)
|
||||
@ -71,20 +73,19 @@ std::string makeRegexpPatternFromGlobs(const std::string & initial_str_with_glob
|
||||
}
|
||||
oss_for_replacing << escaped_with_globs.substr(current_index);
|
||||
std::string almost_res = oss_for_replacing.str();
|
||||
std::ostringstream oss_final_processing;
|
||||
oss_final_processing.exceptions(std::ios::failbit);
|
||||
WriteBufferFromOwnString buf_final_processing;
|
||||
for (const auto & letter : almost_res)
|
||||
{
|
||||
if ((letter == '?') || (letter == '*'))
|
||||
{
|
||||
oss_final_processing << "[^/]"; /// '?' is any symbol except '/'
|
||||
buf_final_processing << "[^/]"; /// '?' is any symbol except '/'
|
||||
if (letter == '?')
|
||||
continue;
|
||||
}
|
||||
if ((letter == '.') || (letter == '{') || (letter == '}'))
|
||||
oss_final_processing << '\\';
|
||||
oss_final_processing << letter;
|
||||
buf_final_processing << '\\';
|
||||
buf_final_processing << letter;
|
||||
}
|
||||
return oss_final_processing.str();
|
||||
return buf_final_processing.str();
|
||||
}
|
||||
}
|
||||
|
||||
@ -9,7 +9,8 @@ using namespace DB;
|
||||
|
||||
TEST(Common, getMultipleValuesFromConfig)
|
||||
{
|
||||
std::istringstream xml_isteam(R"END(<?xml version="1.0"?>
|
||||
std::istringstream // STYLE_CHECK_ALLOW_STD_STRING_STREAM
|
||||
xml_isteam(R"END(<?xml version="1.0"?>
|
||||
<yandex>
|
||||
<first_level>
|
||||
<second_level>0</second_level>
|
||||
|
||||
@ -102,7 +102,8 @@ TEST(Common, SensitiveDataMasker)
|
||||
EXPECT_EQ(maskerbad.wipeSensitiveData(x), 0);
|
||||
|
||||
{
|
||||
std::istringstream xml_isteam(R"END(<?xml version="1.0"?>
|
||||
std::istringstream // STYLE_CHECK_ALLOW_STD_STRING_STREAM
|
||||
xml_isteam(R"END(<?xml version="1.0"?>
|
||||
<clickhouse>
|
||||
<query_masking_rules>
|
||||
<rule>
|
||||
@ -152,7 +153,8 @@ TEST(Common, SensitiveDataMasker)
|
||||
|
||||
try
|
||||
{
|
||||
std::istringstream xml_isteam_bad(R"END(<?xml version="1.0"?>
|
||||
std::istringstream // STYLE_CHECK_ALLOW_STD_STRING_STREAM
|
||||
xml_isteam_bad(R"END(<?xml version="1.0"?>
|
||||
<clickhouse>
|
||||
<query_masking_rules>
|
||||
<rule>
|
||||
@ -181,7 +183,8 @@ TEST(Common, SensitiveDataMasker)
|
||||
|
||||
try
|
||||
{
|
||||
std::istringstream xml_isteam_bad(R"END(<?xml version="1.0"?>
|
||||
std::istringstream // STYLE_CHECK_ALLOW_STD_STRING_STREAM
|
||||
xml_isteam_bad(R"END(<?xml version="1.0"?>
|
||||
<clickhouse>
|
||||
<query_masking_rules>
|
||||
<rule><name>test</name></rule>
|
||||
@ -203,7 +206,8 @@ TEST(Common, SensitiveDataMasker)
|
||||
|
||||
try
|
||||
{
|
||||
std::istringstream xml_isteam_bad(R"END(<?xml version="1.0"?>
|
||||
std::istringstream // STYLE_CHECK_ALLOW_STD_STRING_STREAM
|
||||
xml_isteam_bad(R"END(<?xml version="1.0"?>
|
||||
<clickhouse>
|
||||
<query_masking_rules>
|
||||
<rule><name>test</name><regexp>())(</regexp></rule>
|
||||
|
||||
@ -12,6 +12,7 @@
|
||||
#include <IO/BufferWithOwnMemory.h>
|
||||
#include <Compression/CompressionInfo.h>
|
||||
#include <IO/WriteHelpers.h>
|
||||
#include <IO/Operators.h>
|
||||
|
||||
|
||||
namespace ProfileEvents
|
||||
@ -42,7 +43,7 @@ static void validateChecksum(char * data, size_t size, const Checksum expected_c
|
||||
if (expected_checksum == calculated_checksum)
|
||||
return;
|
||||
|
||||
std::stringstream message;
|
||||
WriteBufferFromOwnString message;
|
||||
|
||||
/// TODO mess up of endianness in error message.
|
||||
message << "Checksum doesn't match: corrupted data."
|
||||
@ -50,7 +51,16 @@ static void validateChecksum(char * data, size_t size, const Checksum expected_c
|
||||
+ ". Actual: " + getHexUIntLowercase(calculated_checksum.first) + getHexUIntLowercase(calculated_checksum.second)
|
||||
+ ". Size of compressed block: " + toString(size);
|
||||
|
||||
const char * message_hardware_failure = "This is most likely due to hardware failure. If you receive broken data over network and the error does not repeat every time, this can be caused by bad RAM on network interface controller or bad controller itself or bad RAM on network switches or bad CPU on network switches (look at the logs on related network switches; note that TCP checksums don't help) or bad RAM on host (look at dmesg or kern.log for enormous amount of EDAC errors, ECC-related reports, Machine Check Exceptions, mcelog; note that ECC memory can fail if the number of errors is huge) or bad CPU on host. If you read data from disk, this can be caused by disk bit rott. This exception protects ClickHouse from data corruption due to hardware failures.";
|
||||
const char * message_hardware_failure = "This is most likely due to hardware failure. "
|
||||
"If you receive broken data over network and the error does not repeat every time, "
|
||||
"this can be caused by bad RAM on network interface controller or bad controller itself "
|
||||
"or bad RAM on network switches or bad CPU on network switches "
|
||||
"(look at the logs on related network switches; note that TCP checksums don't help) "
|
||||
"or bad RAM on host (look at dmesg or kern.log for enormous amount of EDAC errors, "
|
||||
"ECC-related reports, Machine Check Exceptions, mcelog; note that ECC memory can fail "
|
||||
"if the number of errors is huge) or bad CPU on host. If you read data from disk, "
|
||||
"this can be caused by disk bit rott. This exception protects ClickHouse "
|
||||
"from data corruption due to hardware failures.";
|
||||
|
||||
auto flip_bit = [](char * buf, size_t pos)
|
||||
{
|
||||
|
||||
@ -51,10 +51,7 @@ int main(int, char **)
|
||||
|
||||
if (x != i)
|
||||
{
|
||||
std::stringstream s;
|
||||
s.exceptions(std::ios::failbit);
|
||||
s << "Failed!, read: " << x << ", expected: " << i;
|
||||
throw DB::Exception(s.str(), 0);
|
||||
throw DB::Exception(0, "Failed!, read: {}, expected: {}", x, i);
|
||||
}
|
||||
}
|
||||
stopwatch.stop();
|
||||
|
||||
@ -1,5 +1,4 @@
|
||||
#include <Core/MySQL/IMySQLReadPacket.h>
|
||||
#include <sstream>
|
||||
#include <IO/MySQLPacketPayloadReadBuffer.h>
|
||||
#include <IO/LimitReadBuffer.h>
|
||||
|
||||
@ -21,10 +20,9 @@ void IMySQLReadPacket::readPayload(ReadBuffer & in, uint8_t & sequence_id)
|
||||
readPayloadImpl(payload);
|
||||
if (!payload.eof())
|
||||
{
|
||||
std::stringstream tmp;
|
||||
tmp.exceptions(std::ios::failbit);
|
||||
tmp << "Packet payload is not fully read. Stopped after " << payload.count() << " bytes, while " << payload.available() << " bytes are in buffer.";
|
||||
throw Exception(tmp.str(), ErrorCodes::UNKNOWN_PACKET_FROM_CLIENT);
|
||||
throw Exception(ErrorCodes::UNKNOWN_PACKET_FROM_CLIENT,
|
||||
"Packet payload is not fully read. Stopped after {} bytes, while {} bytes are in buffer.",
|
||||
payload.count(), payload.available());
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@ -1,10 +1,14 @@
|
||||
#include <Core/MySQL/IMySQLWritePacket.h>
|
||||
#include <IO/MySQLPacketPayloadWriteBuffer.h>
|
||||
#include <sstream>
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
namespace ErrorCodes
|
||||
{
|
||||
extern const int LOGICAL_ERROR;
|
||||
}
|
||||
|
||||
namespace MySQLProtocol
|
||||
{
|
||||
|
||||
@ -15,10 +19,8 @@ void IMySQLWritePacket::writePayload(WriteBuffer & buffer, uint8_t & sequence_id
|
||||
buf.next();
|
||||
if (buf.remainingPayloadSize())
|
||||
{
|
||||
std::stringstream ss;
|
||||
ss.exceptions(std::ios::failbit);
|
||||
ss << "Incomplete payload. Written " << getPayloadSize() - buf.remainingPayloadSize() << " bytes, expected " << getPayloadSize() << " bytes.";
|
||||
throw Exception(ss.str(), 0);
|
||||
throw Exception(ErrorCodes::LOGICAL_ERROR, "Incomplete payload. Written {} bytes, expected {} bytes.",
|
||||
getPayloadSize() - buf.remainingPayloadSize(), getPayloadSize());
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@ -4,6 +4,7 @@
|
||||
#include <IO/ReadBufferFromString.h>
|
||||
#include <IO/MySQLBinlogEventReadBuffer.h>
|
||||
#include <IO/ReadHelpers.h>
|
||||
#include <IO/Operators.h>
|
||||
#include <common/DateLUT.h>
|
||||
#include <Common/FieldVisitors.h>
|
||||
#include <Core/MySQL/PacketsGeneric.h>
|
||||
@ -35,15 +36,15 @@ namespace MySQLReplication
|
||||
payload.readStrict(reinterpret_cast<char *>(&flags), 2);
|
||||
}
|
||||
|
||||
void EventHeader::dump(std::ostream & out) const
|
||||
void EventHeader::dump(WriteBuffer & out) const
|
||||
{
|
||||
out << "\n=== " << to_string(this->type) << " ===" << std::endl;
|
||||
out << "Timestamp: " << this->timestamp << std::endl;
|
||||
out << "Event Type: " << this->type << std::endl;
|
||||
out << "Server ID: " << this->server_id << std::endl;
|
||||
out << "Event Size: " << this->event_size << std::endl;
|
||||
out << "Log Pos: " << this->log_pos << std::endl;
|
||||
out << "Flags: " << this->flags << std::endl;
|
||||
out << "\n=== " << to_string(this->type) << " ===" << '\n';
|
||||
out << "Timestamp: " << this->timestamp << '\n';
|
||||
out << "Event Type: " << to_string(this->type) << '\n';
|
||||
out << "Server ID: " << this->server_id << '\n';
|
||||
out << "Event Size: " << this->event_size << '\n';
|
||||
out << "Log Pos: " << this->log_pos << '\n';
|
||||
out << "Flags: " << this->flags << '\n';
|
||||
}
|
||||
|
||||
/// https://dev.mysql.com/doc/internals/en/format-description-event.html
|
||||
@ -60,13 +61,13 @@ namespace MySQLReplication
|
||||
readStringUntilEOF(event_type_header_length, payload);
|
||||
}
|
||||
|
||||
void FormatDescriptionEvent::dump(std::ostream & out) const
|
||||
void FormatDescriptionEvent::dump(WriteBuffer & out) const
|
||||
{
|
||||
header.dump(out);
|
||||
out << "Binlog Version: " << this->binlog_version << std::endl;
|
||||
out << "Server Version: " << this->server_version << std::endl;
|
||||
out << "Create Timestamp: " << this->create_timestamp << std::endl;
|
||||
out << "Event Header Len: " << std::to_string(this->event_header_length) << std::endl;
|
||||
out << "Binlog Version: " << this->binlog_version << '\n';
|
||||
out << "Server Version: " << this->server_version << '\n';
|
||||
out << "Create Timestamp: " << this->create_timestamp << '\n';
|
||||
out << "Event Header Len: " << std::to_string(this->event_header_length) << '\n';
|
||||
}
|
||||
|
||||
/// https://dev.mysql.com/doc/internals/en/rotate-event.html
|
||||
@ -76,11 +77,11 @@ namespace MySQLReplication
|
||||
readStringUntilEOF(next_binlog, payload);
|
||||
}
|
||||
|
||||
void RotateEvent::dump(std::ostream & out) const
|
||||
void RotateEvent::dump(WriteBuffer & out) const
|
||||
{
|
||||
header.dump(out);
|
||||
out << "Position: " << this->position << std::endl;
|
||||
out << "Next Binlog: " << this->next_binlog << std::endl;
|
||||
out << "Position: " << this->position << '\n';
|
||||
out << "Next Binlog: " << this->next_binlog << '\n';
|
||||
}
|
||||
|
||||
/// https://dev.mysql.com/doc/internals/en/query-event.html
|
||||
@ -116,24 +117,24 @@ namespace MySQLReplication
|
||||
}
|
||||
}
|
||||
|
||||
void QueryEvent::dump(std::ostream & out) const
|
||||
void QueryEvent::dump(WriteBuffer & out) const
|
||||
{
|
||||
header.dump(out);
|
||||
out << "Thread ID: " << this->thread_id << std::endl;
|
||||
out << "Execution Time: " << this->exec_time << std::endl;
|
||||
out << "Schema Len: " << std::to_string(this->schema_len) << std::endl;
|
||||
out << "Error Code: " << this->error_code << std::endl;
|
||||
out << "Status Len: " << this->status_len << std::endl;
|
||||
out << "Schema: " << this->schema << std::endl;
|
||||
out << "Query: " << this->query << std::endl;
|
||||
out << "Thread ID: " << this->thread_id << '\n';
|
||||
out << "Execution Time: " << this->exec_time << '\n';
|
||||
out << "Schema Len: " << std::to_string(this->schema_len) << '\n';
|
||||
out << "Error Code: " << this->error_code << '\n';
|
||||
out << "Status Len: " << this->status_len << '\n';
|
||||
out << "Schema: " << this->schema << '\n';
|
||||
out << "Query: " << this->query << '\n';
|
||||
}
|
||||
|
||||
void XIDEvent::parseImpl(ReadBuffer & payload) { payload.readStrict(reinterpret_cast<char *>(&xid), 8); }
|
||||
|
||||
void XIDEvent::dump(std::ostream & out) const
|
||||
void XIDEvent::dump(WriteBuffer & out) const
|
||||
{
|
||||
header.dump(out);
|
||||
out << "XID: " << this->xid << std::endl;
|
||||
out << "XID: " << this->xid << '\n';
|
||||
}
|
||||
|
||||
void TableMapEvent::parseImpl(ReadBuffer & payload)
|
||||
@ -238,21 +239,23 @@ namespace MySQLReplication
|
||||
}
|
||||
}
|
||||
|
||||
void TableMapEvent::dump(std::ostream & out) const
|
||||
void TableMapEvent::dump(WriteBuffer & out) const
|
||||
{
|
||||
header.dump(out);
|
||||
out << "Table ID: " << this->table_id << std::endl;
|
||||
out << "Flags: " << this->flags << std::endl;
|
||||
out << "Schema Len: " << std::to_string(this->schema_len) << std::endl;
|
||||
out << "Schema: " << this->schema << std::endl;
|
||||
out << "Table Len: " << std::to_string(this->table_len) << std::endl;
|
||||
out << "Table: " << this->table << std::endl;
|
||||
out << "Column Count: " << this->column_count << std::endl;
|
||||
out << "Table ID: " << this->table_id << '\n';
|
||||
out << "Flags: " << this->flags << '\n';
|
||||
out << "Schema Len: " << std::to_string(this->schema_len) << '\n';
|
||||
out << "Schema: " << this->schema << '\n';
|
||||
out << "Table Len: " << std::to_string(this->table_len) << '\n';
|
||||
out << "Table: " << this->table << '\n';
|
||||
out << "Column Count: " << this->column_count << '\n';
|
||||
for (auto i = 0U; i < column_count; i++)
|
||||
{
|
||||
out << "Column Type [" << i << "]: " << std::to_string(column_type[i]) << ", Meta: " << column_meta[i] << std::endl;
|
||||
out << "Column Type [" << i << "]: " << std::to_string(column_type[i]) << ", Meta: " << column_meta[i] << '\n';
|
||||
}
|
||||
out << "Null Bitmap: " << this->null_bitmap << std::endl;
|
||||
String bitmap_str;
|
||||
boost::to_string(this->null_bitmap, bitmap_str);
|
||||
out << "Null Bitmap: " << bitmap_str << '\n';
|
||||
}
|
||||
|
||||
void RowsEvent::parseImpl(ReadBuffer & payload)
|
||||
@ -631,16 +634,16 @@ namespace MySQLReplication
|
||||
rows.push_back(row);
|
||||
}
|
||||
|
||||
void RowsEvent::dump(std::ostream & out) const
|
||||
void RowsEvent::dump(WriteBuffer & out) const
|
||||
{
|
||||
FieldVisitorToString to_string;
|
||||
|
||||
header.dump(out);
|
||||
out << "Schema: " << this->schema << std::endl;
|
||||
out << "Table: " << this->table << std::endl;
|
||||
out << "Schema: " << this->schema << '\n';
|
||||
out << "Table: " << this->table << '\n';
|
||||
for (auto i = 0U; i < rows.size(); i++)
|
||||
{
|
||||
out << "Row[" << i << "]: " << applyVisitor(to_string, rows[i]) << std::endl;
|
||||
out << "Row[" << i << "]: " << applyVisitor(to_string, rows[i]) << '\n';
|
||||
}
|
||||
}
|
||||
|
||||
@ -663,22 +666,22 @@ namespace MySQLReplication
|
||||
payload.ignoreAll();
|
||||
}
|
||||
|
||||
void GTIDEvent::dump(std::ostream & out) const
|
||||
void GTIDEvent::dump(WriteBuffer & out) const
|
||||
{
|
||||
WriteBufferFromOwnString ws;
|
||||
writeUUIDText(gtid.uuid, ws);
|
||||
auto gtid_next = ws.str() + ":" + std::to_string(gtid.seq_no);
|
||||
|
||||
header.dump(out);
|
||||
out << "GTID Next: " << gtid_next << std::endl;
|
||||
out << "GTID Next: " << gtid_next << '\n';
|
||||
}
|
||||
|
||||
void DryRunEvent::parseImpl(ReadBuffer & payload) { payload.ignoreAll(); }
|
||||
|
||||
void DryRunEvent::dump(std::ostream & out) const
|
||||
void DryRunEvent::dump(WriteBuffer & out) const
|
||||
{
|
||||
header.dump(out);
|
||||
out << "[DryRun Event]" << std::endl;
|
||||
out << "[DryRun Event]" << '\n';
|
||||
}
|
||||
|
||||
/// Update binlog name/position/gtid based on the event type.
|
||||
@ -716,12 +719,12 @@ namespace MySQLReplication
|
||||
gtid_sets.parse(gtid_sets_);
|
||||
}
|
||||
|
||||
void Position::dump(std::ostream & out) const
|
||||
void Position::dump(WriteBuffer & out) const
|
||||
{
|
||||
out << "\n=== Binlog Position ===" << std::endl;
|
||||
out << "Binlog: " << this->binlog_name << std::endl;
|
||||
out << "Position: " << this->binlog_pos << std::endl;
|
||||
out << "GTIDSets: " << this->gtid_sets.toString() << std::endl;
|
||||
out << "\n=== Binlog Position ===" << '\n';
|
||||
out << "Binlog: " << this->binlog_name << '\n';
|
||||
out << "Position: " << this->binlog_pos << '\n';
|
||||
out << "GTIDSets: " << this->gtid_sets.toString() << '\n';
|
||||
}
|
||||
|
||||
void MySQLFlavor::readPayloadImpl(ReadBuffer & payload)
|
||||
|
||||
@ -309,7 +309,7 @@ namespace MySQLReplication
|
||||
UInt16 flags;
|
||||
|
||||
EventHeader() : timestamp(0), server_id(0), event_size(0), log_pos(0), flags(0) { }
|
||||
void dump(std::ostream & out) const;
|
||||
void dump(WriteBuffer & out) const;
|
||||
void parse(ReadBuffer & payload);
|
||||
};
|
||||
|
||||
@ -321,7 +321,7 @@ namespace MySQLReplication
|
||||
EventBase(EventHeader && header_) : header(std::move(header_)) {}
|
||||
|
||||
virtual ~EventBase() = default;
|
||||
virtual void dump(std::ostream & out) const = 0;
|
||||
virtual void dump(WriteBuffer & out) const = 0;
|
||||
virtual void parseEvent(ReadBuffer & payload) { parseImpl(payload); }
|
||||
virtual MySQLEventType type() const { return MYSQL_UNHANDLED_EVENT; }
|
||||
|
||||
@ -344,7 +344,7 @@ namespace MySQLReplication
|
||||
UInt8 event_header_length;
|
||||
String event_type_header_length;
|
||||
|
||||
void dump(std::ostream & out) const override;
|
||||
void dump(WriteBuffer & out) const override;
|
||||
void parseImpl(ReadBuffer & payload) override;
|
||||
|
||||
private:
|
||||
@ -358,7 +358,7 @@ namespace MySQLReplication
|
||||
String next_binlog;
|
||||
|
||||
RotateEvent(EventHeader && header_) : EventBase(std::move(header_)), position(0) {}
|
||||
void dump(std::ostream & out) const override;
|
||||
void dump(WriteBuffer & out) const override;
|
||||
|
||||
protected:
|
||||
void parseImpl(ReadBuffer & payload) override;
|
||||
@ -389,7 +389,7 @@ namespace MySQLReplication
|
||||
{
|
||||
}
|
||||
|
||||
void dump(std::ostream & out) const override;
|
||||
void dump(WriteBuffer & out) const override;
|
||||
MySQLEventType type() const override { return MYSQL_QUERY_EVENT; }
|
||||
|
||||
protected:
|
||||
@ -404,7 +404,7 @@ namespace MySQLReplication
|
||||
protected:
|
||||
UInt64 xid;
|
||||
|
||||
void dump(std::ostream & out) const override;
|
||||
void dump(WriteBuffer & out) const override;
|
||||
void parseImpl(ReadBuffer & payload) override;
|
||||
};
|
||||
|
||||
@ -423,7 +423,7 @@ namespace MySQLReplication
|
||||
Bitmap null_bitmap;
|
||||
|
||||
TableMapEvent(EventHeader && header_) : EventBase(std::move(header_)), table_id(0), flags(0), schema_len(0), table_len(0), column_count(0) {}
|
||||
void dump(std::ostream & out) const override;
|
||||
void dump(WriteBuffer & out) const override;
|
||||
|
||||
protected:
|
||||
void parseImpl(ReadBuffer & payload) override;
|
||||
@ -445,7 +445,7 @@ namespace MySQLReplication
|
||||
table = table_map->table;
|
||||
}
|
||||
|
||||
void dump(std::ostream & out) const override;
|
||||
void dump(WriteBuffer & out) const override;
|
||||
|
||||
protected:
|
||||
UInt64 table_id;
|
||||
@ -489,7 +489,7 @@ namespace MySQLReplication
|
||||
GTID gtid;
|
||||
|
||||
GTIDEvent(EventHeader && header_) : EventBase(std::move(header_)), commit_flag(0) {}
|
||||
void dump(std::ostream & out) const override;
|
||||
void dump(WriteBuffer & out) const override;
|
||||
|
||||
protected:
|
||||
void parseImpl(ReadBuffer & payload) override;
|
||||
@ -499,7 +499,7 @@ namespace MySQLReplication
|
||||
{
|
||||
public:
|
||||
DryRunEvent(EventHeader && header_) : EventBase(std::move(header_)) {}
|
||||
void dump(std::ostream & out) const override;
|
||||
void dump(WriteBuffer & out) const override;
|
||||
|
||||
protected:
|
||||
void parseImpl(ReadBuffer & payload) override;
|
||||
@ -515,7 +515,7 @@ namespace MySQLReplication
|
||||
Position() : binlog_pos(0) { }
|
||||
void update(BinlogEventPtr event);
|
||||
void update(UInt64 binlog_pos_, const String & binlog_name_, const String & gtid_sets_);
|
||||
void dump(std::ostream & out) const;
|
||||
void dump(WriteBuffer & out) const;
|
||||
};
|
||||
|
||||
class IFlavor : public MySQLProtocol::IMySQLReadPacket
|
||||
|
||||
@ -60,10 +60,7 @@ struct SortColumnDescription
|
||||
|
||||
std::string dump() const
|
||||
{
|
||||
std::stringstream ss;
|
||||
ss.exceptions(std::ios::failbit);
|
||||
ss << column_name << ":" << column_number << ":dir " << direction << "nulls " << nulls_direction;
|
||||
return ss.str();
|
||||
return fmt::format("{}:{}:dir {}nulls ", column_name, column_number, direction, nulls_direction);
|
||||
}
|
||||
};
|
||||
|
||||
|
||||
@ -7,6 +7,7 @@
|
||||
#include <Core/MySQL/PacketsProtocolText.h>
|
||||
#include <IO/ReadBufferFromString.h>
|
||||
#include <IO/WriteBufferFromString.h>
|
||||
#include <IO/WriteBufferFromOStream.h>
|
||||
|
||||
#include <boost/program_options.hpp>
|
||||
|
||||
@ -329,6 +330,8 @@ int main(int argc, char ** argv)
|
||||
slave.connect();
|
||||
slave.startBinlogDumpGTID(slave_id, replicate_db, gtid_sets);
|
||||
|
||||
WriteBufferFromOStream cerr(std::cerr);
|
||||
|
||||
/// Read one binlog event on by one.
|
||||
while (true)
|
||||
{
|
||||
@ -337,40 +340,40 @@ int main(int argc, char ** argv)
|
||||
{
|
||||
case MYSQL_QUERY_EVENT: {
|
||||
auto binlog_event = std::static_pointer_cast<QueryEvent>(event);
|
||||
binlog_event->dump(std::cerr);
|
||||
binlog_event->dump(cerr);
|
||||
|
||||
Position pos = slave.getPosition();
|
||||
pos.dump(std::cerr);
|
||||
pos.dump(cerr);
|
||||
break;
|
||||
}
|
||||
case MYSQL_WRITE_ROWS_EVENT: {
|
||||
auto binlog_event = std::static_pointer_cast<WriteRowsEvent>(event);
|
||||
binlog_event->dump(std::cerr);
|
||||
binlog_event->dump(cerr);
|
||||
|
||||
Position pos = slave.getPosition();
|
||||
pos.dump(std::cerr);
|
||||
pos.dump(cerr);
|
||||
break;
|
||||
}
|
||||
case MYSQL_UPDATE_ROWS_EVENT: {
|
||||
auto binlog_event = std::static_pointer_cast<UpdateRowsEvent>(event);
|
||||
binlog_event->dump(std::cerr);
|
||||
binlog_event->dump(cerr);
|
||||
|
||||
Position pos = slave.getPosition();
|
||||
pos.dump(std::cerr);
|
||||
pos.dump(cerr);
|
||||
break;
|
||||
}
|
||||
case MYSQL_DELETE_ROWS_EVENT: {
|
||||
auto binlog_event = std::static_pointer_cast<DeleteRowsEvent>(event);
|
||||
binlog_event->dump(std::cerr);
|
||||
binlog_event->dump(cerr);
|
||||
|
||||
Position pos = slave.getPosition();
|
||||
pos.dump(std::cerr);
|
||||
pos.dump(cerr);
|
||||
break;
|
||||
}
|
||||
default:
|
||||
if (event->header.type != MySQLReplication::EventType::HEARTBEAT_EVENT)
|
||||
{
|
||||
event->dump(std::cerr);
|
||||
event->dump(cerr);
|
||||
}
|
||||
break;
|
||||
}
|
||||
|
||||
@ -59,15 +59,10 @@ void CheckConstraintsBlockOutputStream::write(const Block & block)
|
||||
/// Is violated.
|
||||
if (!value)
|
||||
{
|
||||
std::stringstream exception_message;
|
||||
exception_message.exceptions(std::ios::failbit);
|
||||
|
||||
exception_message << "Constraint " << backQuote(constraint_ptr->name)
|
||||
<< " for table " << table_id.getNameForLogs()
|
||||
<< " is violated, because it is a constant expression returning 0."
|
||||
<< " It is most likely an error in table definition.";
|
||||
|
||||
throw Exception{exception_message.str(), ErrorCodes::VIOLATED_CONSTRAINT};
|
||||
throw Exception(ErrorCodes::VIOLATED_CONSTRAINT,
|
||||
"Constraint {} for table {} is violated, because it is a constant expression returning 0. "
|
||||
"It is most likely an error in table definition.",
|
||||
backQuote(constraint_ptr->name), table_id.getNameForLogs());
|
||||
}
|
||||
}
|
||||
else
|
||||
@ -87,28 +82,27 @@ void CheckConstraintsBlockOutputStream::write(const Block & block)
|
||||
|
||||
Names related_columns = constraint_expr->getRequiredColumns();
|
||||
|
||||
std::stringstream exception_message;
|
||||
exception_message.exceptions(std::ios::failbit);
|
||||
|
||||
exception_message << "Constraint " << backQuote(constraint_ptr->name)
|
||||
<< " for table " << table_id.getNameForLogs()
|
||||
<< " is violated at row " << (rows_written + row_idx + 1)
|
||||
<< ". Expression: (" << serializeAST(*(constraint_ptr->expr), true) << ")"
|
||||
<< ". Column values";
|
||||
|
||||
bool first = true;
|
||||
String column_values_msg;
|
||||
constexpr size_t approx_bytes_for_col = 32;
|
||||
column_values_msg.reserve(approx_bytes_for_col * related_columns.size());
|
||||
for (const auto & name : related_columns)
|
||||
{
|
||||
const IColumn & column = *block.getByName(name).column;
|
||||
assert(row_idx < column.size());
|
||||
|
||||
exception_message << (first ? ": " : ", ")
|
||||
<< backQuoteIfNeed(name) << " = " << applyVisitor(FieldVisitorToString(), column[row_idx]);
|
||||
|
||||
if (!first)
|
||||
column_values_msg.append(", ");
|
||||
column_values_msg.append(backQuoteIfNeed(name));
|
||||
column_values_msg.append(" = ");
|
||||
column_values_msg.append(applyVisitor(FieldVisitorToString(), column[row_idx]));
|
||||
first = false;
|
||||
}
|
||||
|
||||
throw Exception{exception_message.str(), ErrorCodes::VIOLATED_CONSTRAINT};
|
||||
throw Exception(ErrorCodes::VIOLATED_CONSTRAINT,
|
||||
"Constraint {} for table {} is violated at row {}. Expression: ({}). Column values: {}",
|
||||
backQuote(constraint_ptr->name), table_id.getNameForLogs(), rows_written + row_idx + 1,
|
||||
serializeAST(*(constraint_ptr->expr), true), column_values_msg);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@ -4,7 +4,8 @@
|
||||
#include <Interpreters/ProcessList.h>
|
||||
#include <Access/EnabledQuota.h>
|
||||
#include <Common/CurrentThread.h>
|
||||
#include <common/sleep.h>
|
||||
#include <IO/WriteBufferFromString.h>
|
||||
#include <IO/Operators.h>
|
||||
|
||||
namespace ProfileEvents
|
||||
{
|
||||
@ -359,8 +360,7 @@ Block IBlockInputStream::getExtremes()
|
||||
|
||||
String IBlockInputStream::getTreeID() const
|
||||
{
|
||||
std::stringstream s;
|
||||
s.exceptions(std::ios::failbit);
|
||||
WriteBufferFromOwnString s;
|
||||
s << getName();
|
||||
|
||||
if (!children.empty())
|
||||
@ -399,13 +399,13 @@ size_t IBlockInputStream::checkDepthImpl(size_t max_depth, size_t level) const
|
||||
}
|
||||
|
||||
|
||||
void IBlockInputStream::dumpTree(std::ostream & ostr, size_t indent, size_t multiplier) const
|
||||
void IBlockInputStream::dumpTree(WriteBuffer & ostr, size_t indent, size_t multiplier) const
|
||||
{
|
||||
ostr << String(indent, ' ') << getName();
|
||||
if (multiplier > 1)
|
||||
ostr << " × " << multiplier;
|
||||
//ostr << ": " << getHeader().dumpStructure();
|
||||
ostr << std::endl;
|
||||
ostr << '\n';
|
||||
++indent;
|
||||
|
||||
/// If the subtree is repeated several times, then we output it once with the multiplier.
|
||||
|
||||
@ -95,7 +95,7 @@ public:
|
||||
virtual void readSuffix();
|
||||
|
||||
/// Must be called before `read()` and `readPrefix()`.
|
||||
void dumpTree(std::ostream & ostr, size_t indent = 0, size_t multiplier = 1) const;
|
||||
void dumpTree(WriteBuffer & ostr, size_t indent = 0, size_t multiplier = 1) const;
|
||||
|
||||
/** Check the depth of the pipeline.
|
||||
* If max_depth is specified and the `depth` is greater - throw an exception.
|
||||
|
||||
@ -1,6 +1,4 @@
|
||||
#include <queue>
|
||||
#include <iomanip>
|
||||
#include <sstream>
|
||||
|
||||
#include <common/logger_useful.h>
|
||||
|
||||
|
||||
@ -1,4 +1,3 @@
|
||||
#include <sstream>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
|
||||
@ -32,8 +32,7 @@ static const std::vector<String> supported_functions{"any", "anyLast", "min",
|
||||
|
||||
String DataTypeCustomSimpleAggregateFunction::getName() const
|
||||
{
|
||||
std::stringstream stream;
|
||||
stream.exceptions(std::ios::failbit);
|
||||
WriteBufferFromOwnString stream;
|
||||
stream << "SimpleAggregateFunction(" << function->getName();
|
||||
|
||||
if (!parameters.empty())
|
||||
|
||||
@ -29,10 +29,7 @@ namespace ErrorCodes
|
||||
template <typename T>
|
||||
std::string DataTypeDecimal<T>::doGetName() const
|
||||
{
|
||||
std::stringstream ss;
|
||||
ss.exceptions(std::ios::failbit);
|
||||
ss << "Decimal(" << this->precision << ", " << this->scale << ")";
|
||||
return ss.str();
|
||||
return fmt::format("Decimal({}, {})", this->precision, this->scale);
|
||||
}
|
||||
|
||||
|
||||
|
||||
@ -26,7 +26,7 @@ static auto typeFromString(const std::string & str)
|
||||
|
||||
static auto typesFromString(const std::string & str)
|
||||
{
|
||||
std::istringstream data_types_stream(str);
|
||||
std::istringstream data_types_stream(str); // STYLE_CHECK_ALLOW_STD_STRING_STREAM
|
||||
DataTypes data_types;
|
||||
std::string data_type;
|
||||
while (data_types_stream >> data_type)
|
||||
|
||||
@ -94,10 +94,9 @@ String getObjectDefinitionFromCreateQuery(const ASTPtr & query)
|
||||
|
||||
if (!create)
|
||||
{
|
||||
std::ostringstream query_stream;
|
||||
query_stream.exceptions(std::ios::failbit);
|
||||
formatAST(*query, query_stream, true);
|
||||
throw Exception("Query '" + query_stream.str() + "' is not CREATE query", ErrorCodes::LOGICAL_ERROR);
|
||||
WriteBufferFromOwnString query_buf;
|
||||
formatAST(*query, query_buf, true);
|
||||
throw Exception(ErrorCodes::LOGICAL_ERROR, "Query '{}' is not CREATE query", query_buf.str());
|
||||
}
|
||||
|
||||
if (!create->is_dictionary)
|
||||
@ -121,11 +120,10 @@ String getObjectDefinitionFromCreateQuery(const ASTPtr & query)
|
||||
if (create->uuid != UUIDHelpers::Nil)
|
||||
create->table = TABLE_WITH_UUID_NAME_PLACEHOLDER;
|
||||
|
||||
std::ostringstream statement_stream;
|
||||
statement_stream.exceptions(std::ios::failbit);
|
||||
formatAST(*create, statement_stream, false);
|
||||
statement_stream << '\n';
|
||||
return statement_stream.str();
|
||||
WriteBufferFromOwnString statement_buf;
|
||||
formatAST(*create, statement_buf, false);
|
||||
writeChar('\n', statement_buf);
|
||||
return statement_buf.str();
|
||||
}
|
||||
|
||||
DatabaseOnDisk::DatabaseOnDisk(
|
||||
|
||||
@ -127,8 +127,7 @@ static String checkVariableAndGetVersion(const mysqlxx::Pool::Entry & connection
|
||||
}
|
||||
|
||||
bool first = true;
|
||||
std::stringstream error_message;
|
||||
error_message.exceptions(std::ios::failbit);
|
||||
WriteBufferFromOwnString error_message;
|
||||
error_message << "Illegal MySQL variables, the MaterializeMySQL engine requires ";
|
||||
for (const auto & [variable_name, variable_error_message] : variables_error_message)
|
||||
{
|
||||
@ -239,8 +238,7 @@ static inline BlockOutputStreamPtr getTableOutput(const String & database_name,
|
||||
{
|
||||
const StoragePtr & storage = DatabaseCatalog::instance().getTable(StorageID(database_name, table_name), query_context);
|
||||
|
||||
std::stringstream insert_columns_str;
|
||||
insert_columns_str.exceptions(std::ios::failbit);
|
||||
WriteBufferFromOwnString insert_columns_str;
|
||||
const StorageInMemoryMetadata & storage_metadata = storage->getInMemoryMetadata();
|
||||
const ColumnsDescription & storage_columns = storage_metadata.getColumns();
|
||||
const NamesAndTypesList & insert_columns_names = insert_materialized ? storage_columns.getAllPhysical() : storage_columns.getOrdinary();
|
||||
@ -331,10 +329,9 @@ std::optional<MaterializeMetadata> MaterializeMySQLSyncThread::prepareSynchroniz
|
||||
|
||||
const auto & position_message = [&]()
|
||||
{
|
||||
std::stringstream ss;
|
||||
ss.exceptions(std::ios::failbit);
|
||||
position.dump(ss);
|
||||
return ss.str();
|
||||
WriteBufferFromOwnString buf;
|
||||
position.dump(buf);
|
||||
return buf.str();
|
||||
};
|
||||
LOG_INFO(log, "MySQL dump database position: \n {}", position_message());
|
||||
}
|
||||
@ -374,10 +371,9 @@ void MaterializeMySQLSyncThread::flushBuffersData(Buffers & buffers, Materialize
|
||||
|
||||
const auto & position_message = [&]()
|
||||
{
|
||||
std::stringstream ss;
|
||||
ss.exceptions(std::ios::failbit);
|
||||
client.getPosition().dump(ss);
|
||||
return ss.str();
|
||||
WriteBufferFromOwnString buf;
|
||||
client.getPosition().dump(buf);
|
||||
return buf.str();
|
||||
};
|
||||
LOG_INFO(log, "MySQL executed position: \n {}", position_message());
|
||||
}
|
||||
@ -632,10 +628,9 @@ void MaterializeMySQLSyncThread::onEvent(Buffers & buffers, const BinlogEventPtr
|
||||
{
|
||||
const auto & dump_event_message = [&]()
|
||||
{
|
||||
std::stringstream ss;
|
||||
ss.exceptions(std::ios::failbit);
|
||||
receive_event->dump(ss);
|
||||
return ss.str();
|
||||
WriteBufferFromOwnString buf;
|
||||
receive_event->dump(buf);
|
||||
return buf.str();
|
||||
};
|
||||
|
||||
LOG_DEBUG(log, "Skip MySQL event: \n {}", dump_event_message());
|
||||
|
||||
@ -4,6 +4,7 @@
|
||||
#include <DataTypes/DataTypeNullable.h>
|
||||
#include <Formats/FormatSettings.h>
|
||||
#include <IO/WriteHelpers.h>
|
||||
#include <IO/Operators.h>
|
||||
#include <Common/StringUtils/StringUtils.h>
|
||||
|
||||
#include <numeric>
|
||||
@ -230,8 +231,7 @@ std::string DictionaryStructure::getKeyDescription() const
|
||||
if (id)
|
||||
return "UInt64";
|
||||
|
||||
std::ostringstream out;
|
||||
out.exceptions(std::ios::failbit);
|
||||
WriteBufferFromOwnString out;
|
||||
|
||||
out << '(';
|
||||
|
||||
|
||||
@ -18,7 +18,7 @@ static bool registered = false;
|
||||
static std::string configurationToString(const DictionaryConfigurationPtr & config)
|
||||
{
|
||||
const Poco::Util::XMLConfiguration * xml_config = dynamic_cast<const Poco::Util::XMLConfiguration *>(config.get());
|
||||
std::ostringstream oss;
|
||||
std::ostringstream oss; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
|
||||
oss.exceptions(std::ios::failbit);
|
||||
xml_config->save(oss);
|
||||
return oss.str();
|
||||
|
||||
@ -3,7 +3,6 @@
|
||||
#if !defined(ARCADIA_BUILD) && USE_STATS
|
||||
|
||||
#include <math.h>
|
||||
#include <sstream>
|
||||
|
||||
#include <DataTypes/DataTypeString.h>
|
||||
#include <Columns/ColumnString.h>
|
||||
|
||||
@ -1,11 +1,13 @@
|
||||
#include <IO/CompressionMethod.h>
|
||||
|
||||
#include <IO/ReadBuffer.h>
|
||||
#include <IO/WriteBuffer.h>
|
||||
#include <IO/ZlibInflatingReadBuffer.h>
|
||||
#include <IO/ZlibDeflatingWriteBuffer.h>
|
||||
#include <IO/BrotliReadBuffer.h>
|
||||
#include <IO/BrotliWriteBuffer.h>
|
||||
#include <IO/LZMADeflatingWriteBuffer.h>
|
||||
#include <IO/LZMAInflatingReadBuffer.h>
|
||||
#include <IO/ReadBuffer.h>
|
||||
#include <IO/WriteBuffer.h>
|
||||
#include <IO/ZlibDeflatingWriteBuffer.h>
|
||||
#include <IO/ZlibInflatingReadBuffer.h>
|
||||
|
||||
#if !defined(ARCADIA_BUILD)
|
||||
# include <Common/config.h>
|
||||
@ -14,7 +16,6 @@
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
namespace ErrorCodes
|
||||
{
|
||||
extern const int NOT_IMPLEMENTED;
|
||||
@ -25,10 +26,16 @@ std::string toContentEncodingName(CompressionMethod method)
|
||||
{
|
||||
switch (method)
|
||||
{
|
||||
case CompressionMethod::Gzip: return "gzip";
|
||||
case CompressionMethod::Zlib: return "deflate";
|
||||
case CompressionMethod::Brotli: return "br";
|
||||
case CompressionMethod::None: return "";
|
||||
case CompressionMethod::Gzip:
|
||||
return "gzip";
|
||||
case CompressionMethod::Zlib:
|
||||
return "deflate";
|
||||
case CompressionMethod::Brotli:
|
||||
return "br";
|
||||
case CompressionMethod::Xz:
|
||||
return "xz";
|
||||
case CompressionMethod::None:
|
||||
return "";
|
||||
}
|
||||
__builtin_unreachable();
|
||||
}
|
||||
@ -52,20 +59,19 @@ CompressionMethod chooseCompressionMethod(const std::string & path, const std::s
|
||||
return CompressionMethod::Zlib;
|
||||
if (*method_str == "brotli" || *method_str == "br")
|
||||
return CompressionMethod::Brotli;
|
||||
if (*method_str == "LZMA" || *method_str == "xz")
|
||||
return CompressionMethod::Xz;
|
||||
if (hint.empty() || hint == "auto" || hint == "none")
|
||||
return CompressionMethod::None;
|
||||
|
||||
throw Exception("Unknown compression method " + hint + ". Only 'auto', 'none', 'gzip', 'br' are supported as compression methods",
|
||||
throw Exception(
|
||||
"Unknown compression method " + hint + ". Only 'auto', 'none', 'gzip', 'br', 'xz' are supported as compression methods",
|
||||
ErrorCodes::NOT_IMPLEMENTED);
|
||||
}
|
||||
|
||||
|
||||
std::unique_ptr<ReadBuffer> wrapReadBufferWithCompressionMethod(
|
||||
std::unique_ptr<ReadBuffer> nested,
|
||||
CompressionMethod method,
|
||||
size_t buf_size,
|
||||
char * existing_memory,
|
||||
size_t alignment)
|
||||
std::unique_ptr<ReadBuffer> nested, CompressionMethod method, size_t buf_size, char * existing_memory, size_t alignment)
|
||||
{
|
||||
if (method == CompressionMethod::Gzip || method == CompressionMethod::Zlib)
|
||||
return std::make_unique<ZlibInflatingReadBuffer>(std::move(nested), method, buf_size, existing_memory, alignment);
|
||||
@ -73,6 +79,8 @@ std::unique_ptr<ReadBuffer> wrapReadBufferWithCompressionMethod(
|
||||
if (method == CompressionMethod::Brotli)
|
||||
return std::make_unique<BrotliReadBuffer>(std::move(nested), buf_size, existing_memory, alignment);
|
||||
#endif
|
||||
if (method == CompressionMethod::Xz)
|
||||
return std::make_unique<LZMAInflatingReadBuffer>(std::move(nested), buf_size, existing_memory, alignment);
|
||||
|
||||
if (method == CompressionMethod::None)
|
||||
return nested;
|
||||
@ -82,12 +90,7 @@ std::unique_ptr<ReadBuffer> wrapReadBufferWithCompressionMethod(
|
||||
|
||||
|
||||
std::unique_ptr<WriteBuffer> wrapWriteBufferWithCompressionMethod(
|
||||
std::unique_ptr<WriteBuffer> nested,
|
||||
CompressionMethod method,
|
||||
int level,
|
||||
size_t buf_size,
|
||||
char * existing_memory,
|
||||
size_t alignment)
|
||||
std::unique_ptr<WriteBuffer> nested, CompressionMethod method, int level, size_t buf_size, char * existing_memory, size_t alignment)
|
||||
{
|
||||
if (method == DB::CompressionMethod::Gzip || method == CompressionMethod::Zlib)
|
||||
return std::make_unique<ZlibDeflatingWriteBuffer>(std::move(nested), method, level, buf_size, existing_memory, alignment);
|
||||
@ -96,6 +99,8 @@ std::unique_ptr<WriteBuffer> wrapWriteBufferWithCompressionMethod(
|
||||
if (method == DB::CompressionMethod::Brotli)
|
||||
return std::make_unique<BrotliWriteBuffer>(std::move(nested), level, buf_size, existing_memory, alignment);
|
||||
#endif
|
||||
if (method == CompressionMethod::Xz)
|
||||
return std::make_unique<LZMADeflatingWriteBuffer>(std::move(nested), level, buf_size, existing_memory, alignment);
|
||||
|
||||
if (method == CompressionMethod::None)
|
||||
return nested;
|
||||
|
||||
@ -1,14 +1,13 @@
|
||||
#pragma once
|
||||
|
||||
#include <string>
|
||||
#include <memory>
|
||||
#include <string>
|
||||
|
||||
#include <Core/Defines.h>
|
||||
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
class ReadBuffer;
|
||||
class WriteBuffer;
|
||||
|
||||
@ -26,6 +25,9 @@ enum class CompressionMethod
|
||||
/// DEFLATE compression with zlib header and Adler32 checksum.
|
||||
/// This option corresponds to HTTP Content-Encoding: deflate.
|
||||
Zlib,
|
||||
/// LZMA2-based content compression
|
||||
/// This option corresponds to HTTP Content-Encoding: xz
|
||||
Xz,
|
||||
Brotli
|
||||
};
|
||||
|
||||
|
||||
@ -239,7 +239,7 @@ void assertResponseIsOk(const Poco::Net::HTTPRequest & request, Poco::Net::HTTPR
|
||||
|
||||
if (!(status == Poco::Net::HTTPResponse::HTTP_OK || (isRedirect(status) && allow_redirects)))
|
||||
{
|
||||
std::stringstream error_message;
|
||||
std::stringstream error_message; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
|
||||
error_message.exceptions(std::ios::failbit);
|
||||
error_message << "Received error from remote server " << request.getURI() << ". HTTP status code: " << status << " "
|
||||
<< response.getReason() << ", body: " << istr.rdbuf();
|
||||
|
||||
125
src/IO/LZMADeflatingWriteBuffer.cpp
Normal file
125
src/IO/LZMADeflatingWriteBuffer.cpp
Normal file
@ -0,0 +1,125 @@
|
||||
#include <IO/LZMADeflatingWriteBuffer.h>
|
||||
|
||||
|
||||
namespace DB
|
||||
{
|
||||
namespace ErrorCodes
|
||||
{
|
||||
extern const int LZMA_STREAM_ENCODER_FAILED;
|
||||
}
|
||||
|
||||
LZMADeflatingWriteBuffer::LZMADeflatingWriteBuffer(
|
||||
std::unique_ptr<WriteBuffer> out_, int compression_level, size_t buf_size, char * existing_memory, size_t alignment)
|
||||
: BufferWithOwnMemory<WriteBuffer>(buf_size, existing_memory, alignment), out(std::move(out_))
|
||||
{
|
||||
|
||||
lstr = LZMA_STREAM_INIT;
|
||||
lstr.allocator = nullptr;
|
||||
lstr.next_in = nullptr;
|
||||
lstr.avail_in = 0;
|
||||
lstr.next_out = nullptr;
|
||||
lstr.avail_out = 0;
|
||||
|
||||
// options for further compression
|
||||
lzma_options_lzma opt_lzma2;
|
||||
if (lzma_lzma_preset(&opt_lzma2, compression_level))
|
||||
throw Exception(ErrorCodes::LZMA_STREAM_ENCODER_FAILED, "lzma preset failed: lzma version: {}", LZMA_VERSION_STRING);
|
||||
|
||||
|
||||
// LZMA_FILTER_X86 -
|
||||
// LZMA2 - codec for *.xz files compression; LZMA is not suitable for this purpose
|
||||
// VLI - variable length integer (in *.xz most integers encoded as VLI)
|
||||
// LZMA_VLI_UNKNOWN (UINT64_MAX) - VLI value to denote that the value is unknown
|
||||
lzma_filter filters[] = {
|
||||
{.id = LZMA_FILTER_X86, .options = nullptr},
|
||||
{.id = LZMA_FILTER_LZMA2, .options = &opt_lzma2},
|
||||
{.id = LZMA_VLI_UNKNOWN, .options = nullptr},
|
||||
};
|
||||
lzma_ret ret = lzma_stream_encoder(&lstr, filters, LZMA_CHECK_CRC64);
|
||||
|
||||
if (ret != LZMA_OK)
|
||||
throw Exception(
|
||||
ErrorCodes::LZMA_STREAM_ENCODER_FAILED,
|
||||
"lzma stream encoder init failed: error code: {} lzma version: {}",
|
||||
ret,
|
||||
LZMA_VERSION_STRING);
|
||||
}
|
||||
|
||||
LZMADeflatingWriteBuffer::~LZMADeflatingWriteBuffer()
|
||||
{
|
||||
try
|
||||
{
|
||||
finish();
|
||||
|
||||
lzma_end(&lstr);
|
||||
}
|
||||
catch (...)
|
||||
{
|
||||
tryLogCurrentException(__PRETTY_FUNCTION__);
|
||||
}
|
||||
}
|
||||
|
||||
void LZMADeflatingWriteBuffer::nextImpl()
|
||||
{
|
||||
if (!offset())
|
||||
return;
|
||||
|
||||
lstr.next_in = reinterpret_cast<unsigned char *>(working_buffer.begin());
|
||||
lstr.avail_in = offset();
|
||||
|
||||
lzma_action action = LZMA_RUN;
|
||||
do
|
||||
{
|
||||
out->nextIfAtEnd();
|
||||
lstr.next_out = reinterpret_cast<unsigned char *>(out->position());
|
||||
lstr.avail_out = out->buffer().end() - out->position();
|
||||
|
||||
lzma_ret ret = lzma_code(&lstr, action);
|
||||
out->position() = out->buffer().end() - lstr.avail_out;
|
||||
|
||||
if (ret == LZMA_STREAM_END)
|
||||
return;
|
||||
|
||||
if (ret != LZMA_OK)
|
||||
throw Exception(
|
||||
ErrorCodes::LZMA_STREAM_ENCODER_FAILED,
|
||||
"lzma stream encoding failed: error code: {}; lzma_version: {}",
|
||||
ret,
|
||||
LZMA_VERSION_STRING);
|
||||
|
||||
} while (lstr.avail_in > 0 || lstr.avail_out == 0);
|
||||
}
|
||||
|
||||
|
||||
void LZMADeflatingWriteBuffer::finish()
|
||||
{
|
||||
if (finished)
|
||||
return;
|
||||
|
||||
next();
|
||||
|
||||
do
|
||||
{
|
||||
out->nextIfAtEnd();
|
||||
lstr.next_out = reinterpret_cast<unsigned char *>(out->position());
|
||||
lstr.avail_out = out->buffer().end() - out->position();
|
||||
|
||||
lzma_ret ret = lzma_code(&lstr, LZMA_FINISH);
|
||||
out->position() = out->buffer().end() - lstr.avail_out;
|
||||
|
||||
if (ret == LZMA_STREAM_END)
|
||||
{
|
||||
finished = true;
|
||||
return;
|
||||
}
|
||||
|
||||
if (ret != LZMA_OK)
|
||||
throw Exception(
|
||||
ErrorCodes::LZMA_STREAM_ENCODER_FAILED,
|
||||
"lzma stream encoding failed: error code: {}; lzma version: {}",
|
||||
ret,
|
||||
LZMA_VERSION_STRING);
|
||||
|
||||
} while (lstr.avail_out == 0);
|
||||
}
|
||||
}
|
||||
32
src/IO/LZMADeflatingWriteBuffer.h
Normal file
32
src/IO/LZMADeflatingWriteBuffer.h
Normal file
@ -0,0 +1,32 @@
|
||||
#pragma once
|
||||
|
||||
#include <IO/BufferWithOwnMemory.h>
|
||||
#include <IO/WriteBuffer.h>
|
||||
|
||||
#include <lzma.h>
|
||||
|
||||
namespace DB
|
||||
{
|
||||
/// Performs compression using lzma library and writes compressed data to out_ WriteBuffer.
|
||||
class LZMADeflatingWriteBuffer : public BufferWithOwnMemory<WriteBuffer>
|
||||
{
|
||||
public:
|
||||
LZMADeflatingWriteBuffer(
|
||||
std::unique_ptr<WriteBuffer> out_,
|
||||
int compression_level,
|
||||
size_t buf_size = DBMS_DEFAULT_BUFFER_SIZE,
|
||||
char * existing_memory = nullptr,
|
||||
size_t alignment = 0);
|
||||
|
||||
void finish();
|
||||
|
||||
~LZMADeflatingWriteBuffer() override;
|
||||
|
||||
private:
|
||||
void nextImpl() override;
|
||||
|
||||
std::unique_ptr<WriteBuffer> out;
|
||||
lzma_stream lstr;
|
||||
bool finished = false;
|
||||
};
|
||||
}
|
||||
89
src/IO/LZMAInflatingReadBuffer.cpp
Normal file
89
src/IO/LZMAInflatingReadBuffer.cpp
Normal file
@ -0,0 +1,89 @@
|
||||
#include <IO/LZMAInflatingReadBuffer.h>
|
||||
|
||||
namespace DB
|
||||
{
|
||||
namespace ErrorCodes
|
||||
{
|
||||
extern const int LZMA_STREAM_DECODER_FAILED;
|
||||
}
|
||||
LZMAInflatingReadBuffer::LZMAInflatingReadBuffer(std::unique_ptr<ReadBuffer> in_, size_t buf_size, char * existing_memory, size_t alignment)
|
||||
: BufferWithOwnMemory<ReadBuffer>(buf_size, existing_memory, alignment), in(std::move(in_)), eof(false)
|
||||
{
|
||||
lstr = LZMA_STREAM_INIT;
|
||||
lstr.allocator = nullptr;
|
||||
lstr.next_in = nullptr;
|
||||
lstr.avail_in = 0;
|
||||
lstr.next_out = nullptr;
|
||||
lstr.avail_out = 0;
|
||||
|
||||
// 500 mb
|
||||
uint64_t memlimit = 500ULL << 20;
|
||||
|
||||
lzma_ret ret = lzma_stream_decoder(&lstr, memlimit, LZMA_CONCATENATED);
|
||||
// lzma does not provide api for converting error code to string unlike zlib
|
||||
if (ret != LZMA_OK)
|
||||
throw Exception(
|
||||
ErrorCodes::LZMA_STREAM_DECODER_FAILED,
|
||||
"lzma_stream_decoder initialization failed: error code: {}; lzma version: {}",
|
||||
ret,
|
||||
LZMA_VERSION_STRING);
|
||||
}
|
||||
|
||||
LZMAInflatingReadBuffer::~LZMAInflatingReadBuffer()
|
||||
{
|
||||
lzma_end(&lstr);
|
||||
}
|
||||
|
||||
bool LZMAInflatingReadBuffer::nextImpl()
|
||||
{
|
||||
if (eof)
|
||||
return false;
|
||||
|
||||
lzma_action action = LZMA_RUN;
|
||||
|
||||
if (!lstr.avail_in)
|
||||
{
|
||||
in->nextIfAtEnd();
|
||||
lstr.next_in = reinterpret_cast<unsigned char *>(in->position());
|
||||
lstr.avail_in = in->buffer().end() - in->position();
|
||||
}
|
||||
|
||||
if (in->eof())
|
||||
{
|
||||
action = LZMA_FINISH;
|
||||
}
|
||||
|
||||
lstr.next_out = reinterpret_cast<unsigned char *>(internal_buffer.begin());
|
||||
lstr.avail_out = internal_buffer.size();
|
||||
|
||||
lzma_ret ret = lzma_code(&lstr, action);
|
||||
in->position() = in->buffer().end() - lstr.avail_in;
|
||||
working_buffer.resize(internal_buffer.size() - lstr.avail_out);
|
||||
|
||||
if (ret == LZMA_STREAM_END)
|
||||
{
|
||||
if (in->eof())
|
||||
{
|
||||
eof = true;
|
||||
return working_buffer.size() != 0;
|
||||
}
|
||||
else
|
||||
{
|
||||
throw Exception(
|
||||
ErrorCodes::LZMA_STREAM_DECODER_FAILED,
|
||||
"lzma decoder finished, but input stream has not exceeded: error code: {}; lzma version: {}",
|
||||
ret,
|
||||
LZMA_VERSION_STRING);
|
||||
}
|
||||
}
|
||||
|
||||
if (ret != LZMA_OK)
|
||||
throw Exception(
|
||||
ErrorCodes::LZMA_STREAM_DECODER_FAILED,
|
||||
"lzma_stream_decoder failed: error code: error codeL {}; lzma version: {}",
|
||||
ret,
|
||||
LZMA_VERSION_STRING);
|
||||
|
||||
return true;
|
||||
}
|
||||
}
|
||||
33
src/IO/LZMAInflatingReadBuffer.h
Normal file
33
src/IO/LZMAInflatingReadBuffer.h
Normal file
@ -0,0 +1,33 @@
|
||||
#pragma once
|
||||
|
||||
#include <IO/BufferWithOwnMemory.h>
|
||||
#include <IO/ReadBuffer.h>
|
||||
|
||||
#include <lzma.h>
|
||||
|
||||
namespace DB
|
||||
{
|
||||
namespace ErrorCodes
|
||||
{
|
||||
}
|
||||
|
||||
class LZMAInflatingReadBuffer : public BufferWithOwnMemory<ReadBuffer>
|
||||
{
|
||||
public:
|
||||
LZMAInflatingReadBuffer(
|
||||
std::unique_ptr<ReadBuffer> in_,
|
||||
size_t buf_size = DBMS_DEFAULT_BUFFER_SIZE,
|
||||
char * existing_memory = nullptr,
|
||||
size_t alignment = 0);
|
||||
|
||||
~LZMAInflatingReadBuffer() override;
|
||||
|
||||
private:
|
||||
bool nextImpl() override;
|
||||
|
||||
std::unique_ptr<ReadBuffer> in;
|
||||
lzma_stream lstr;
|
||||
|
||||
bool eof;
|
||||
};
|
||||
}
|
||||
@ -1,5 +1,4 @@
|
||||
#include <IO/MySQLPacketPayloadReadBuffer.h>
|
||||
#include <sstream>
|
||||
|
||||
namespace DB
|
||||
{
|
||||
|
||||
@ -45,7 +45,9 @@ struct BinaryManipReadBuffer : std::reference_wrapper<ReadBuffer> { usin
|
||||
template <typename T> WriteBuffer & operator<< (WriteBuffer & buf, const T & x) { writeText(x, buf); return buf; }
|
||||
/// If you do not use the manipulators, the string is displayed without an escape, as is.
|
||||
template <> inline WriteBuffer & operator<< (WriteBuffer & buf, const String & x) { writeString(x, buf); return buf; }
|
||||
template <> inline WriteBuffer & operator<< (WriteBuffer & buf, const std::string_view & x) { writeString(StringRef(x), buf); return buf; }
|
||||
template <> inline WriteBuffer & operator<< (WriteBuffer & buf, const char & x) { writeChar(x, buf); return buf; }
|
||||
template <> inline WriteBuffer & operator<< (WriteBuffer & buf, const pcg32_fast & x) { PcgSerializer::serializePcg32(x, buf); return buf; }
|
||||
|
||||
inline WriteBuffer & operator<< (WriteBuffer & buf, const char * x) { writeCString(x, buf); return buf; }
|
||||
|
||||
@ -73,6 +75,7 @@ inline WriteBuffer & operator<< (WriteBuffer & buf, FlushManip) { buf.next(); re
|
||||
template <typename T> ReadBuffer & operator>> (ReadBuffer & buf, T & x) { readText(x, buf); return buf; }
|
||||
template <> inline ReadBuffer & operator>> (ReadBuffer & buf, String & x) { readString(x, buf); return buf; }
|
||||
template <> inline ReadBuffer & operator>> (ReadBuffer & buf, char & x) { readChar(x, buf); return buf; }
|
||||
template <> inline ReadBuffer & operator>> (ReadBuffer & buf, pcg32_fast & x) { PcgDeserializer::deserializePcg32(x, buf); return buf; }
|
||||
|
||||
/// If you specify a string literal for reading, this will mean - make sure there is a sequence of bytes and skip it.
|
||||
inline ReadBuffer & operator>> (ReadBuffer & buf, const char * x) { assertString(x, buf); return buf; }
|
||||
|
||||
@ -53,6 +53,7 @@ namespace ErrorCodes
|
||||
extern const int CANNOT_READ_ARRAY_FROM_TEXT;
|
||||
extern const int CANNOT_PARSE_NUMBER;
|
||||
extern const int ILLEGAL_TYPE_OF_ARGUMENT;
|
||||
extern const int INCORRECT_DATA;
|
||||
}
|
||||
|
||||
/// Helper functions for formatted input.
|
||||
@ -1233,4 +1234,23 @@ void saveUpToPosition(ReadBuffer & in, Memory<> & memory, char * current);
|
||||
*/
|
||||
bool loadAtPosition(ReadBuffer & in, Memory<> & memory, char * & current);
|
||||
|
||||
|
||||
struct PcgDeserializer
|
||||
{
|
||||
static void deserializePcg32(const pcg32_fast & rng, ReadBuffer & buf)
|
||||
{
|
||||
decltype(rng.state_) multiplier, increment, state;
|
||||
readText(multiplier, buf);
|
||||
assertChar(' ', buf);
|
||||
readText(increment, buf);
|
||||
assertChar(' ', buf);
|
||||
readText(state, buf);
|
||||
|
||||
if (multiplier != rng.multiplier())
|
||||
throw Exception(ErrorCodes::INCORRECT_DATA, "Incorrect multiplier in pcg32: expected {}, got {}", rng.multiplier(), multiplier);
|
||||
if (increment != rng.increment())
|
||||
throw Exception(ErrorCodes::INCORRECT_DATA, "Incorrect increment in pcg32: expected {}, got {}", rng.increment(), increment);
|
||||
}
|
||||
};
|
||||
|
||||
}
|
||||
|
||||
@ -7,6 +7,8 @@
|
||||
#include <utility>
|
||||
#include <IO/HTTPCommon.h>
|
||||
#include <IO/S3/PocoHTTPResponseStream.h>
|
||||
#include <IO/WriteBufferFromString.h>
|
||||
#include <IO/Operators.h>
|
||||
#include <Common/Stopwatch.h>
|
||||
#include <aws/core/http/HttpRequest.h>
|
||||
#include <aws/core/http/HttpResponse.h>
|
||||
@ -247,8 +249,7 @@ void PocoHTTPClient::makeRequestInternal(
|
||||
response->SetResponseCode(static_cast<Aws::Http::HttpResponseCode>(status_code));
|
||||
response->SetContentType(poco_response.getContentType());
|
||||
|
||||
std::stringstream headers_ss;
|
||||
headers_ss.exceptions(std::ios::failbit);
|
||||
WriteBufferFromOwnString headers_ss;
|
||||
for (const auto & [header_name, header_value] : poco_response)
|
||||
{
|
||||
response->AddHeader(header_name, header_value);
|
||||
|
||||
@ -36,8 +36,10 @@ private:
|
||||
throw Exception("WriteBufferFromVector is finished", ErrorCodes::CANNOT_WRITE_AFTER_END_OF_BUFFER);
|
||||
|
||||
size_t old_size = vector.size();
|
||||
/// pos may not be equal to vector.data() + old_size, because WriteBuffer::next() can be used to flush data
|
||||
size_t pos_offset = pos - reinterpret_cast<Position>(vector.data());
|
||||
vector.resize(old_size * size_multiplier);
|
||||
internal_buffer = Buffer(reinterpret_cast<Position>(vector.data() + old_size), reinterpret_cast<Position>(vector.data() + vector.size()));
|
||||
internal_buffer = Buffer(reinterpret_cast<Position>(vector.data() + pos_offset), reinterpret_cast<Position>(vector.data() + vector.size()));
|
||||
working_buffer = internal_buffer;
|
||||
}
|
||||
|
||||
|
||||
@ -89,4 +89,12 @@ void writeProbablyBackQuotedStringMySQL(const StringRef & s, WriteBuffer & buf)
|
||||
writeProbablyQuotedStringImpl(s, buf, [](const StringRef & s_, WriteBuffer & buf_) { return writeBackQuotedStringMySQL(s_, buf_); });
|
||||
}
|
||||
|
||||
void writePointerHex(const void * ptr, WriteBuffer & buf)
|
||||
{
|
||||
writeString("0x", buf);
|
||||
char hex_str[2 * sizeof(ptr)];
|
||||
writeHexUIntLowercase(reinterpret_cast<uintptr_t>(ptr), hex_str);
|
||||
buf.write(hex_str, 2 * sizeof(ptr));
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@ -1093,4 +1093,18 @@ writeBinaryBigEndian(T x, WriteBuffer & buf) /// Assuming little endian archi
|
||||
writePODBinary(x, buf);
|
||||
}
|
||||
|
||||
struct PcgSerializer
|
||||
{
|
||||
static void serializePcg32(const pcg32_fast & rng, WriteBuffer & buf)
|
||||
{
|
||||
writeText(rng.multiplier(), buf);
|
||||
writeChar(' ', buf);
|
||||
writeText(rng.increment(), buf);
|
||||
writeChar(' ', buf);
|
||||
writeText(rng.state_, buf);
|
||||
}
|
||||
};
|
||||
|
||||
void writePointerHex(const void * ptr, WriteBuffer & buf);
|
||||
|
||||
}
|
||||
|
||||
@ -65,6 +65,9 @@ endif ()
|
||||
add_executable (zlib_buffers zlib_buffers.cpp)
|
||||
target_link_libraries (zlib_buffers PRIVATE clickhouse_common_io)
|
||||
|
||||
add_executable (lzma_buffers lzma_buffers.cpp)
|
||||
target_link_libraries (lzma_buffers PRIVATE clickhouse_common_io)
|
||||
|
||||
add_executable (limit_read_buffer limit_read_buffer.cpp)
|
||||
target_link_libraries (limit_read_buffer PRIVATE clickhouse_common_io)
|
||||
|
||||
|
||||
@ -76,7 +76,7 @@ std::string dumpContents(const T& container,
|
||||
const size_t cols_in_row = 8)
|
||||
|
||||
{
|
||||
std::stringstream sstr;
|
||||
std::stringstream sstr; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
|
||||
sstr.exceptions(std::ios::failbit);
|
||||
dumpBuffer(std::begin(container), std::end(container), &sstr, col_sep, row_sep, cols_in_row);
|
||||
|
||||
|
||||
@ -22,7 +22,7 @@ static void test(size_t data_size)
|
||||
for (size_t read_buffer_block_size : block_sizes)
|
||||
{
|
||||
std::cout << "block size " << read_buffer_block_size << std::endl;
|
||||
std::stringstream io;
|
||||
std::stringstream io; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
|
||||
io.exceptions(std::ios::failbit);
|
||||
DB::WriteBufferFromOStream out_impl(io);
|
||||
DB::HashingWriteBuffer out(out_impl);
|
||||
|
||||
@ -20,7 +20,7 @@ try
|
||||
{
|
||||
using namespace DB;
|
||||
|
||||
std::stringstream s;
|
||||
std::stringstream s; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
|
||||
s.exceptions(std::ios::failbit);
|
||||
|
||||
{
|
||||
|
||||
64
src/IO/tests/lzma_buffers.cpp
Normal file
64
src/IO/tests/lzma_buffers.cpp
Normal file
@ -0,0 +1,64 @@
|
||||
#include <iomanip>
|
||||
#include <iostream>
|
||||
|
||||
#include <IO/LZMADeflatingWriteBuffer.h>
|
||||
#include <IO/LZMAInflatingReadBuffer.h>
|
||||
#include <IO/ReadBufferFromFile.h>
|
||||
#include <IO/ReadHelpers.h>
|
||||
#include <IO/WriteBufferFromFile.h>
|
||||
#include <IO/WriteHelpers.h>
|
||||
#include <Common/Stopwatch.h>
|
||||
|
||||
int main(int, char **)
|
||||
try
|
||||
{
|
||||
std::cout << std::fixed << std::setprecision(2);
|
||||
|
||||
size_t n = 10000000;
|
||||
Stopwatch stopwatch;
|
||||
|
||||
{
|
||||
auto buf
|
||||
= std::make_unique<DB::WriteBufferFromFile>("test_lzma_buffers.xz", DBMS_DEFAULT_BUFFER_SIZE, O_WRONLY | O_CREAT | O_TRUNC);
|
||||
DB::LZMADeflatingWriteBuffer lzma_buf(std::move(buf), /*compression level*/ 3);
|
||||
|
||||
stopwatch.restart();
|
||||
for (size_t i = 0; i < n; ++i)
|
||||
{
|
||||
DB::writeIntText(i, lzma_buf);
|
||||
DB::writeChar('\t', lzma_buf);
|
||||
}
|
||||
lzma_buf.finish();
|
||||
|
||||
stopwatch.stop();
|
||||
|
||||
std::cout << "Writing done. Elapsed: " << stopwatch.elapsedSeconds() << " s."
|
||||
<< ", " << (lzma_buf.count() / stopwatch.elapsedSeconds() / 1000000) << " MB/s" << std::endl;
|
||||
}
|
||||
|
||||
{
|
||||
auto buf = std::make_unique<DB::ReadBufferFromFile>("test_lzma_buffers.xz");
|
||||
DB::LZMAInflatingReadBuffer lzma_buf(std::move(buf));
|
||||
|
||||
stopwatch.restart();
|
||||
for (size_t i = 0; i < n; ++i)
|
||||
{
|
||||
size_t x;
|
||||
DB::readIntText(x, lzma_buf);
|
||||
lzma_buf.ignore();
|
||||
|
||||
if (x != i)
|
||||
throw DB::Exception("Failed!, read: " + std::to_string(x) + ", expected: " + std::to_string(i), 0);
|
||||
}
|
||||
stopwatch.stop();
|
||||
std::cout << "Reading done. Elapsed: " << stopwatch.elapsedSeconds() << " s."
|
||||
<< ", " << (lzma_buf.count() / stopwatch.elapsedSeconds() / 1000000) << " MB/s" << std::endl;
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
catch (const DB::Exception & e)
|
||||
{
|
||||
std::cerr << e.what() << ", " << e.displayText() << std::endl;
|
||||
return 1;
|
||||
}
|
||||
@ -16,7 +16,7 @@ int main(int, char **)
|
||||
DB::String c = "вася пе\tтя";
|
||||
DB::String d = "'xyz\\";
|
||||
|
||||
std::stringstream s;
|
||||
std::stringstream s; // STYLE_CHECK_ALLOW_STD_STRING_STREAM
|
||||
s.exceptions(std::ios::failbit);
|
||||
|
||||
{
|
||||
|
||||
@ -20,6 +20,8 @@ SRCS(
|
||||
HTTPCommon.cpp
|
||||
HashingWriteBuffer.cpp
|
||||
HexWriteBuffer.cpp
|
||||
LZMADeflatingWriteBuffer.cpp
|
||||
LZMAInflatingReadBuffer.cpp
|
||||
LimitReadBuffer.cpp
|
||||
MMapReadBufferFromFile.cpp
|
||||
MMapReadBufferFromFileDescriptor.cpp
|
||||
|
||||
@ -25,7 +25,7 @@ class AddDefaultDatabaseVisitor
|
||||
{
|
||||
public:
|
||||
explicit AddDefaultDatabaseVisitor(
|
||||
const String & database_name_, bool only_replace_current_database_function_ = false, std::ostream * ostr_ = nullptr)
|
||||
const String & database_name_, bool only_replace_current_database_function_ = false, WriteBuffer * ostr_ = nullptr)
|
||||
: database_name(database_name_)
|
||||
, only_replace_current_database_function(only_replace_current_database_function_)
|
||||
, visit_depth(0)
|
||||
@ -66,7 +66,7 @@ private:
|
||||
const String database_name;
|
||||
bool only_replace_current_database_function = false;
|
||||
mutable size_t visit_depth;
|
||||
std::ostream * ostr;
|
||||
WriteBuffer * ostr;
|
||||
|
||||
void visit(ASTSelectWithUnionQuery & select, ASTPtr &) const
|
||||
{
|
||||
|
||||
@ -98,10 +98,9 @@ String formattedAST(const ASTPtr & ast)
|
||||
{
|
||||
if (!ast)
|
||||
return {};
|
||||
std::stringstream ss;
|
||||
ss.exceptions(std::ios::failbit);
|
||||
formatAST(*ast, ss, false, true);
|
||||
return ss.str();
|
||||
WriteBufferFromOwnString buf;
|
||||
formatAST(*ast, buf, false, true);
|
||||
return buf.str();
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@ -2005,21 +2005,19 @@ void Context::checkCanBeDropped(const String & database, const String & table, c
|
||||
|
||||
String size_str = formatReadableSizeWithDecimalSuffix(size);
|
||||
String max_size_to_drop_str = formatReadableSizeWithDecimalSuffix(max_size_to_drop);
|
||||
std::stringstream ostr;
|
||||
ostr.exceptions(std::ios::failbit);
|
||||
|
||||
ostr << "Table or Partition in " << backQuoteIfNeed(database) << "." << backQuoteIfNeed(table) << " was not dropped.\n"
|
||||
<< "Reason:\n"
|
||||
<< "1. Size (" << size_str << ") is greater than max_[table/partition]_size_to_drop (" << max_size_to_drop_str << ")\n"
|
||||
<< "2. File '" << force_file.path() << "' intended to force DROP "
|
||||
<< (force_file_exists ? "exists but not writeable (could not be removed)" : "doesn't exist") << "\n";
|
||||
|
||||
ostr << "How to fix this:\n"
|
||||
<< "1. Either increase (or set to zero) max_[table/partition]_size_to_drop in server config\n"
|
||||
<< "2. Either create forcing file " << force_file.path() << " and make sure that ClickHouse has write permission for it.\n"
|
||||
<< "Example:\nsudo touch '" << force_file.path() << "' && sudo chmod 666 '" << force_file.path() << "'";
|
||||
|
||||
throw Exception(ostr.str(), ErrorCodes::TABLE_SIZE_EXCEEDS_MAX_DROP_SIZE_LIMIT);
|
||||
throw Exception(ErrorCodes::TABLE_SIZE_EXCEEDS_MAX_DROP_SIZE_LIMIT,
|
||||
"Table or Partition in {}.{} was not dropped.\nReason:\n"
|
||||
"1. Size ({}) is greater than max_[table/partition]_size_to_drop ({})\n"
|
||||
"2. File '{}' intended to force DROP {}\n"
|
||||
"How to fix this:\n"
|
||||
"1. Either increase (or set to zero) max_[table/partition]_size_to_drop in server config\n",
|
||||
"2. Either create forcing file {} and make sure that ClickHouse has write permission for it.\n"
|
||||
"Example:\nsudo touch '{}' && sudo chmod 666 '{}'",
|
||||
backQuoteIfNeed(database), backQuoteIfNeed(table),
|
||||
size_str, max_size_to_drop_str,
|
||||
force_file.path(), force_file_exists ? "exists but not writeable (could not be removed)" : "doesn't exist",
|
||||
force_file.path(),
|
||||
force_file.path(), force_file.path());
|
||||
}
|
||||
|
||||
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue
Block a user