diff --git a/.gitmodules b/.gitmodules index eaef4cb01ec..61b1fe27a4c 100644 --- a/.gitmodules +++ b/.gitmodules @@ -197,3 +197,6 @@ path = contrib/rocksdb url = https://github.com/facebook/rocksdb branch = v6.11.4 +[submodule "contrib/xz"] + path = contrib/xz + url = https://github.com/xz-mirror/xz diff --git a/CHANGELOG.md b/CHANGELOG.md index 457346aff9a..4474675e9ee 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,122 @@ +### ClickHouse release v20.11.2.1, 2020-11-11 + +#### Backward Incompatible Change + +* If some `profile` was specified in `distributed_ddl` config section, then this profile could overwrite settings of `default` profile on server startup. It's fixed, now settings of distributed DDL queries should not affect global server settings. [#16635](https://github.com/ClickHouse/ClickHouse/pull/16635) ([tavplubix](https://github.com/tavplubix)). +* Restrict to use of non-comparable data types (like `AggregateFunction`) in keys (Sorting key, Primary key, Partition key, and so on). [#16601](https://github.com/ClickHouse/ClickHouse/pull/16601) ([alesapin](https://github.com/alesapin)). +* Remove `ANALYZE` and `AST` queries, and make the setting `enable_debug_queries` obsolete since now it is the part of full featured `EXPLAIN` query. [#16536](https://github.com/ClickHouse/ClickHouse/pull/16536) ([Ivan](https://github.com/abyss7)). +* Aggregate functions `boundingRatio`, `rankCorr`, `retention`, `timeSeriesGroupSum`, `timeSeriesGroupRateSum`, `windowFunnel` were erroneously made case-insensitive. Now their names are made case sensitive as designed. Only functions that are specified in SQL standard or made for compatibility with other DBMS or functions similar to those should be case-insensitive. [#16407](https://github.com/ClickHouse/ClickHouse/pull/16407) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Make `rankCorr` function return nan on insufficient data https://github.com/ClickHouse/ClickHouse/issues/16124. [#16135](https://github.com/ClickHouse/ClickHouse/pull/16135) ([hexiaoting](https://github.com/hexiaoting)). + +#### New Feature + +* Added support of LDAP as a user directory for locally non-existent users. [#12736](https://github.com/ClickHouse/ClickHouse/pull/12736) ([Denis Glazachev](https://github.com/traceon)). +* Add `system.replicated_fetches` table which shows currently running background fetches. [#16428](https://github.com/ClickHouse/ClickHouse/pull/16428) ([alesapin](https://github.com/alesapin)). +* Added setting `date_time_output_format`. [#15845](https://github.com/ClickHouse/ClickHouse/pull/15845) ([Maksim Kita](https://github.com/kitaisreal)). +* Added minimal web UI to ClickHouse. [#16158](https://github.com/ClickHouse/ClickHouse/pull/16158) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Allows to read/write Single protobuf message at once (w/o length-delimiters). [#15199](https://github.com/ClickHouse/ClickHouse/pull/15199) ([filimonov](https://github.com/filimonov)). +* Added initial OpenTelemetry support. ClickHouse now accepts OpenTelemetry traceparent headers over Native and HTTP protocols, and passes them downstream in some cases. The trace spans for executed queries are saved into the `system.opentelemetry_span_log` table. [#14195](https://github.com/ClickHouse/ClickHouse/pull/14195) ([Alexander Kuzmenkov](https://github.com/akuzm)). +* Allow specify primary key in column list of `CREATE TABLE` query. This is needed for compatibility with other SQL dialects. [#15823](https://github.com/ClickHouse/ClickHouse/pull/15823) ([Maksim Kita](https://github.com/kitaisreal)). +* Implement `OFFSET offset_row_count {ROW | ROWS} FETCH {FIRST | NEXT} fetch_row_count {ROW | ROWS} {ONLY | WITH TIES}` in SELECT query with ORDER BY. This is the SQL-standard way to specify `LIMIT`. [#15855](https://github.com/ClickHouse/ClickHouse/pull/15855) ([hexiaoting](https://github.com/hexiaoting)). +* `errorCodeToName` function - return variable name of the error (useful for analyzing query_log and similar). `system.errors` table - shows how many times errors has been happened (respects `system_events_show_zero_values`). [#16438](https://github.com/ClickHouse/ClickHouse/pull/16438) ([Azat Khuzhin](https://github.com/azat)). +* Added function `untuple` which is a special function which can introduce new columns to the SELECT list by expanding a named tuple. [#16242](https://github.com/ClickHouse/ClickHouse/pull/16242) ([Nikolai Kochetov](https://github.com/KochetovNicolai), [Amos Bird](https://github.com/amosbird)). +* Now we can provide identifiers via query parameters. And these parameters can be used as table objects or columns. [#16594](https://github.com/ClickHouse/ClickHouse/pull/16594) ([Amos Bird](https://github.com/amosbird)). +* Added big integers (UInt256, Int128, Int256) and UUID data types support for MergeTree BloomFilter index. Big integers is an experimental feature. [#16642](https://github.com/ClickHouse/ClickHouse/pull/16642) ([Maksim Kita](https://github.com/kitaisreal)). +* Add `farmFingerprint64` function (non-cryptographic string hashing). [#16570](https://github.com/ClickHouse/ClickHouse/pull/16570) ([Jacob Hayes](https://github.com/JacobHayes)). +* Add `log_queries_min_query_duration_ms`, only queries slower then the value of this setting will go to `query_log`/`query_thread_log` (i.e. something like `slow_query_log` in mysql). [#16529](https://github.com/ClickHouse/ClickHouse/pull/16529) ([Azat Khuzhin](https://github.com/azat)). +* Ability to create a docker image on the top of `Alpine`. Uses precompiled binary and glibc components from ubuntu 20.04. [#16479](https://github.com/ClickHouse/ClickHouse/pull/16479) ([filimonov](https://github.com/filimonov)). +* Added `toUUIDOrNull`, `toUUIDOrZero` cast functions. [#16337](https://github.com/ClickHouse/ClickHouse/pull/16337) ([Maksim Kita](https://github.com/kitaisreal)). +* Add `max_concurrent_queries_for_all_users` setting, see [#6636](https://github.com/ClickHouse/ClickHouse/issues/6636) for use cases. [#16154](https://github.com/ClickHouse/ClickHouse/pull/16154) ([nvartolomei](https://github.com/nvartolomei)). +* Add a new option `print_query_id` to clickhouse-client. It helps generate arbitrary strings with the current query id generated by the client. Also print query id in clickhouse-client by default. [#15809](https://github.com/ClickHouse/ClickHouse/pull/15809) ([Amos Bird](https://github.com/amosbird)). +* Add `tid` and `logTrace` functions. This closes [#9434](https://github.com/ClickHouse/ClickHouse/issues/9434). [#15803](https://github.com/ClickHouse/ClickHouse/pull/15803) ([flynn](https://github.com/ucasFL)). +* Add function `formatReadableTimeDelta` that format time delta to human readable string ... [#15497](https://github.com/ClickHouse/ClickHouse/pull/15497) ([Filipe Caixeta](https://github.com/filipecaixeta)). +* Added `disable_merges` option for volumes in multi-disk configuration. [#13956](https://github.com/ClickHouse/ClickHouse/pull/13956) ([Vladimir Chebotarev](https://github.com/excitoon)). + +#### Experimental Feature + +* New functions `encrypt`, `aes_encrypt_mysql`, `decrypt`, `aes_decrypt_mysql`. These functions are working slowly, so we consider it as an experimental feature. [#11844](https://github.com/ClickHouse/ClickHouse/pull/11844) ([Vasily Nemkov](https://github.com/Enmk)). + +#### Bug Fix + +* Mask password in data_path in the `system.distribution_queue`. [#16727](https://github.com/ClickHouse/ClickHouse/pull/16727) ([Azat Khuzhin](https://github.com/azat)). +* Fix `IN` operator over several columns and tuples with enabled `transform_null_in` setting. Fixes [#15310](https://github.com/ClickHouse/ClickHouse/issues/15310). [#16722](https://github.com/ClickHouse/ClickHouse/pull/16722) ([Anton Popov](https://github.com/CurtizJ)). +* The setting `max_parallel_replicas` worked incorrectly if the queried table has no sampling. This fixes [#5733](https://github.com/ClickHouse/ClickHouse/issues/5733). [#16675](https://github.com/ClickHouse/ClickHouse/pull/16675) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Fix optimize_read_in_order/optimize_aggregation_in_order with max_threads > 0 and expression in ORDER BY. [#16637](https://github.com/ClickHouse/ClickHouse/pull/16637) ([Azat Khuzhin](https://github.com/azat)). +* Calculation of `DEFAULT` expressions was involving possible name collisions (that was very unlikely to encounter). This fixes [#9359](https://github.com/ClickHouse/ClickHouse/issues/9359). [#16612](https://github.com/ClickHouse/ClickHouse/pull/16612) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Fix `query_thread_log.query_duration_ms` unit. [#16563](https://github.com/ClickHouse/ClickHouse/pull/16563) ([Azat Khuzhin](https://github.com/azat)). +* Fix a bug when using MySQL Master -> MySQL Slave -> ClickHouse MaterializeMySQL Engine. `MaterializeMySQL` is an experimental feature. [#16504](https://github.com/ClickHouse/ClickHouse/pull/16504) ([TCeason](https://github.com/TCeason)). +* Specifically crafted argument of `round` function with `Decimal` was leading to integer division by zero. This fixes [#13338](https://github.com/ClickHouse/ClickHouse/issues/13338). [#16451](https://github.com/ClickHouse/ClickHouse/pull/16451) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Fix DROP TABLE for Distributed (racy with INSERT). [#16409](https://github.com/ClickHouse/ClickHouse/pull/16409) ([Azat Khuzhin](https://github.com/azat)). +* Fix processing of very large entries in replication queue. Very large entries may appear in ALTER queries if table structure is extremely large (near 1 MB). This fixes [#16307](https://github.com/ClickHouse/ClickHouse/issues/16307). [#16332](https://github.com/ClickHouse/ClickHouse/pull/16332) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Fixed the inconsistent behaviour when a part of return data could be dropped because the set for its filtration wasn't created. [#16308](https://github.com/ClickHouse/ClickHouse/pull/16308) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). +* Fix dictGet in sharding_key (and similar places, i.e. when the function context is stored permanently). [#16205](https://github.com/ClickHouse/ClickHouse/pull/16205) ([Azat Khuzhin](https://github.com/azat)). +* Fix the exception thrown in `clickhouse-local` when trying to execute `OPTIMIZE` command. Fixes [#16076](https://github.com/ClickHouse/ClickHouse/issues/16076). [#16192](https://github.com/ClickHouse/ClickHouse/pull/16192) ([filimonov](https://github.com/filimonov)). +* Fixes [#15780](https://github.com/ClickHouse/ClickHouse/issues/15780) regression, e.g. `indexOf([1, 2, 3], toLowCardinality(1))` now is prohibited but it should not be. [#16038](https://github.com/ClickHouse/ClickHouse/pull/16038) ([Mike](https://github.com/myrrc)). +* Fix bug with MySQL database. When MySQL server used as database engine is down some queries raise Exception, because they try to get tables from disabled server, while it's unnecessary. For example, query `SELECT ... FROM system.parts` should work only with MergeTree tables and don't touch MySQL database at all. [#16032](https://github.com/ClickHouse/ClickHouse/pull/16032) ([Kruglov Pavel](https://github.com/Avogar)). +* Now exception will be thrown when `ALTER MODIFY COLUMN ... DEFAULT ...` has incompatible default with column type. Fixes [#15854](https://github.com/ClickHouse/ClickHouse/issues/15854). [#15858](https://github.com/ClickHouse/ClickHouse/pull/15858) ([alesapin](https://github.com/alesapin)). +* Fixed IPv4CIDRToRange/IPv6CIDRToRange functions to accept const IP-column values. [#15856](https://github.com/ClickHouse/ClickHouse/pull/15856) ([vladimir-golovchenko](https://github.com/vladimir-golovchenko)). + +#### Improvement + +* Treat `INTERVAL '1 hour'` as equivalent to `INTERVAL 1 HOUR`, to be compatible with Postgres and similar. This fixes [#15637](https://github.com/ClickHouse/ClickHouse/issues/15637). [#15978](https://github.com/ClickHouse/ClickHouse/pull/15978) ([flynn](https://github.com/ucasFL)). +* Enable parsing enum values by their numeric ids for CSV, TSV and JSON input formats. [#15685](https://github.com/ClickHouse/ClickHouse/pull/15685) ([vivarum](https://github.com/vivarum)). +* Better read task scheduling for JBOD architecture and `MergeTree` storage. New setting `read_backoff_min_concurrency` which serves as the lower limit to the number of reading threads. [#16423](https://github.com/ClickHouse/ClickHouse/pull/16423) ([Amos Bird](https://github.com/amosbird)). +* Add missing support for `LowCardinality` in `Avro` format. [#16521](https://github.com/ClickHouse/ClickHouse/pull/16521) ([Mike](https://github.com/myrrc)). +* Workaround for use `S3` with nginx server as proxy. Nginx currenty does not accept urls with empty path like `http://domain.com?delete`, but vanilla aws-sdk-cpp produces this kind of urls. This commit uses patched aws-sdk-cpp version, which makes urls with "/" as path in this cases, like `http://domain.com/?delete`. [#16814](https://github.com/ClickHouse/ClickHouse/pull/16814) ([ianton-ru](https://github.com/ianton-ru)). +* Better diagnostics on parse errors in input data. Provide row number on `Cannot read all data` errors. [#16644](https://github.com/ClickHouse/ClickHouse/pull/16644) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Make the behaviour of `minMap` and `maxMap` more desireable. It will not skip zero values in the result. Fixes [#16087](https://github.com/ClickHouse/ClickHouse/issues/16087). [#16631](https://github.com/ClickHouse/ClickHouse/pull/16631) ([Ildus Kurbangaliev](https://github.com/ildus)). +* Better update of ZooKeeper configuration in runtime. [#16630](https://github.com/ClickHouse/ClickHouse/pull/16630) ([sundyli](https://github.com/sundy-li)). +* Apply SETTINGS clause as early as possible. It allows to modify more settings in the query. This closes [#3178](https://github.com/ClickHouse/ClickHouse/issues/3178). [#16619](https://github.com/ClickHouse/ClickHouse/pull/16619) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Now `event_time_microseconds` field stores in Decimal64, not UInt64. [#16617](https://github.com/ClickHouse/ClickHouse/pull/16617) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). +* Now paratmeterized functions can be used in `APPLY` column transformer. [#16589](https://github.com/ClickHouse/ClickHouse/pull/16589) ([Amos Bird](https://github.com/amosbird)). +* Improve scheduling of background task which removes data of dropped tables in `Atomic` databases. `Atomic` databases do not create broken symlink to table data directory if table actually has no data directory. [#16584](https://github.com/ClickHouse/ClickHouse/pull/16584) ([tavplubix](https://github.com/tavplubix)). +* Subqueries in `WITH` section (CTE) can reference previous subqueries in `WITH` section by their name. [#16575](https://github.com/ClickHouse/ClickHouse/pull/16575) ([Amos Bird](https://github.com/amosbird)). +* Add current_database into `system.query_thread_log`. [#16558](https://github.com/ClickHouse/ClickHouse/pull/16558) ([Azat Khuzhin](https://github.com/azat)). +* Allow to fetch parts that are already committed or outdated in the current instance into the detached directory. It's useful when migrating tables from another cluster and having N to 1 shards mapping. It's also consistent with the current fetchPartition implementation. [#16538](https://github.com/ClickHouse/ClickHouse/pull/16538) ([Amos Bird](https://github.com/amosbird)). +* Multiple improvements for `RabbitMQ`: Fixed bug for [#16263](https://github.com/ClickHouse/ClickHouse/issues/16263). Also minimized event loop lifetime. Added more efficient queues setup. [#16426](https://github.com/ClickHouse/ClickHouse/pull/16426) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Fix debug assertion in `quantileDeterministic` function. In previous version it may also transfer up to two times more data over the network. Although no bug existed. This fixes [#15683](https://github.com/ClickHouse/ClickHouse/issues/15683). [#16410](https://github.com/ClickHouse/ClickHouse/pull/16410) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Add `TablesToDropQueueSize` metric. It's equal to number of dropped tables, that are waiting for background data removal. [#16364](https://github.com/ClickHouse/ClickHouse/pull/16364) ([tavplubix](https://github.com/tavplubix)). +* Better diagnostics when client has dropped connection. In previous versions, `Attempt to read after EOF` and `Broken pipe` exceptions were logged in server. In new version, it's information message `Client has dropped the connection, cancel the query.`. [#16329](https://github.com/ClickHouse/ClickHouse/pull/16329) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Add total_rows/total_bytes (from system.tables) support for Set/Join table engines. [#16306](https://github.com/ClickHouse/ClickHouse/pull/16306) ([Azat Khuzhin](https://github.com/azat)). +* Now it's possible to specify `PRIMARY KEY` without `ORDER BY` for MergeTree table engines family. Closes [#15591](https://github.com/ClickHouse/ClickHouse/issues/15591). [#16284](https://github.com/ClickHouse/ClickHouse/pull/16284) ([alesapin](https://github.com/alesapin)). +* If there is no tmp folder in the system (chroot, misconfigutation etc) `clickhouse-local` will create temporary subfolder in the current directory. [#16280](https://github.com/ClickHouse/ClickHouse/pull/16280) ([filimonov](https://github.com/filimonov)). +* Add support for nested data types (like named tuple) as sub-types. Fixes [#15587](https://github.com/ClickHouse/ClickHouse/issues/15587). [#16262](https://github.com/ClickHouse/ClickHouse/pull/16262) ([Ivan](https://github.com/abyss7)). +* Support for `database_atomic_wait_for_drop_and_detach_synchronously`/`NO DELAY`/`SYNC` for `DROP DATABASE`. [#16127](https://github.com/ClickHouse/ClickHouse/pull/16127) ([Azat Khuzhin](https://github.com/azat)). +* Add `allow_nondeterministic_optimize_skip_unused_shards` (to allow non deterministic like `rand()` or `dictGet()` in sharding key). [#16105](https://github.com/ClickHouse/ClickHouse/pull/16105) ([Azat Khuzhin](https://github.com/azat)). +* Fix `memory_profiler_step`/`max_untracked_memory` for queries via HTTP (test included). Fix the issue that adjusting this value globally in xml config does not help either, since those settings are not applied anyway, only default (4MB) value is [used](https://github.com/ClickHouse/ClickHouse/blob/17731245336d8c84f75e4c0894c5797ed7732190/src/Common/ThreadStatus.h#L104). Fix `query_id` for the most root ThreadStatus of the http query (by initializing QueryScope after reading query_id). [#16101](https://github.com/ClickHouse/ClickHouse/pull/16101) ([Azat Khuzhin](https://github.com/azat)). +* Now it's allowed to execute `ALTER ... ON CLUSTER` queries regardless of the `` setting in cluster config. [#16075](https://github.com/ClickHouse/ClickHouse/pull/16075) ([alesapin](https://github.com/alesapin)). +* Fix rare issue when `clickhouse-client` may abort on exit due to loading of suggestions. This fixes [#16035](https://github.com/ClickHouse/ClickHouse/issues/16035). [#16047](https://github.com/ClickHouse/ClickHouse/pull/16047) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Add support of `cache` layout for `Redis` dictionaries with complex key. [#15985](https://github.com/ClickHouse/ClickHouse/pull/15985) ([Anton Popov](https://github.com/CurtizJ)). +* Fix query hang (endless loop) in case of misconfiguration (`connections_with_failover_max_tries` set to 0). [#15876](https://github.com/ClickHouse/ClickHouse/pull/15876) ([Azat Khuzhin](https://github.com/azat)). +* Change level of some log messages from information to debug, so information messages will not appear for every query. This closes [#5293](https://github.com/ClickHouse/ClickHouse/issues/5293). [#15816](https://github.com/ClickHouse/ClickHouse/pull/15816) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Remove `MemoryTrackingInBackground*` metrics to avoid potentially misleading results. This fixes [#15684](https://github.com/ClickHouse/ClickHouse/issues/15684). [#15813](https://github.com/ClickHouse/ClickHouse/pull/15813) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Add reconnects to `zookeeper-dump-tree` tool. [#15711](https://github.com/ClickHouse/ClickHouse/pull/15711) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Allow explicitly specify columns list in `CREATE TABLE table AS table_function(...)` query. Fixes [#9249](https://github.com/ClickHouse/ClickHouse/issues/9249) Fixes [#14214](https://github.com/ClickHouse/ClickHouse/issues/14214). [#14295](https://github.com/ClickHouse/ClickHouse/pull/14295) ([tavplubix](https://github.com/tavplubix)). + +#### Performance Improvement + +* Do not merge parts across partitions in SELECT FINAL. [#15938](https://github.com/ClickHouse/ClickHouse/pull/15938) ([Kruglov Pavel](https://github.com/Avogar)). +* Improve performance of `-OrNull` and `-OrDefault` aggregate functions. [#16661](https://github.com/ClickHouse/ClickHouse/pull/16661) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Improve performance of `quantileMerge`. In previous versions it was obnoxiously slow. This closes [#1463](https://github.com/ClickHouse/ClickHouse/issues/1463). [#16643](https://github.com/ClickHouse/ClickHouse/pull/16643) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Improve performance of logical functions a little. [#16347](https://github.com/ClickHouse/ClickHouse/pull/16347) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Improved performance of merges assignment in MergeTree table engines. Shouldn't be visible for the user. [#16191](https://github.com/ClickHouse/ClickHouse/pull/16191) ([alesapin](https://github.com/alesapin)). +* Speedup hashed/sparse_hashed dictionary loading by preallocating the hash table. [#15454](https://github.com/ClickHouse/ClickHouse/pull/15454) ([Azat Khuzhin](https://github.com/azat)). +* Now trivial count optimization becomes slightly non-trivial. Predicates that contain exact partition expr can be optimized too. This also fixes [#11092](https://github.com/ClickHouse/ClickHouse/issues/11092) which returns wrong count when `max_parallel_replicas > 1`. [#15074](https://github.com/ClickHouse/ClickHouse/pull/15074) ([Amos Bird](https://github.com/amosbird)). + +#### Build/Testing/Packaging Improvement + +* Add flaky check for stateless tests. It will detect potentially flaky functional tests in advance, before they are merged. [#16238](https://github.com/ClickHouse/ClickHouse/pull/16238) ([alesapin](https://github.com/alesapin)). +* Use proper version for `croaring` instead of amalgamation. [#16285](https://github.com/ClickHouse/ClickHouse/pull/16285) ([sundyli](https://github.com/sundy-li)). +* Improve generation of build files for `ya.make` build system (Arcadia). [#16700](https://github.com/ClickHouse/ClickHouse/pull/16700) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Add MySQL BinLog file check tool for `MaterializeMySQL` database engine. `MaterializeMySQL` is an experimental feature. [#16223](https://github.com/ClickHouse/ClickHouse/pull/16223) ([Winter Zhang](https://github.com/zhang2014)). +* Check for executable bit on non-executable files. People often accidentially commit executable files from Windows. [#15843](https://github.com/ClickHouse/ClickHouse/pull/15843) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Check for `#pragma once` in headers. [#15818](https://github.com/ClickHouse/ClickHouse/pull/15818) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Fix illegal code style `&vector[idx]` in libhdfs3. This fixes libcxx debug build. See also https://github.com/ClickHouse-Extras/libhdfs3/pull/8 . [#15815](https://github.com/ClickHouse/ClickHouse/pull/15815) ([Amos Bird](https://github.com/amosbird)). +* Fix build of one miscellaneous example tool on Mac OS. Note that we don't build examples on Mac OS in our CI (we build only ClickHouse binary), so there is zero chance it will not break again. This fixes [#15804](https://github.com/ClickHouse/ClickHouse/issues/15804). [#15808](https://github.com/ClickHouse/ClickHouse/pull/15808) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Simplify Sys/V init script. [#14135](https://github.com/ClickHouse/ClickHouse/pull/14135) ([alexey-milovidov](https://github.com/alexey-milovidov)). +* Added `boost::program_options` to `db_generator` in order to increase its usability. This closes [#15940](https://github.com/ClickHouse/ClickHouse/issues/15940). [#15973](https://github.com/ClickHouse/ClickHouse/pull/15973) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). + + ## ClickHouse release 20.10 ### ClickHouse release v20.10.3.30, 2020-10-28 diff --git a/base/pcg-random/pcg_random.hpp b/base/pcg-random/pcg_random.hpp index d96d5895b31..f7778480c4c 100644 --- a/base/pcg-random/pcg_random.hpp +++ b/base/pcg-random/pcg_random.hpp @@ -113,6 +113,12 @@ #include "pcg_extras.hpp" +namespace DB +{ + struct PcgSerializer; + struct PcgDeserializer; +} + namespace pcg_detail { using namespace pcg_extras; @@ -557,6 +563,9 @@ public: engine& rng); + + friend ::DB::PcgSerializer; + friend ::DB::PcgDeserializer; }; template diff --git a/docs/zh/engines/table-engines/log-family/log.md b/docs/zh/engines/table-engines/log-family/log.md index 4a3855c4bfa..5d3ee38d745 100644 --- a/docs/zh/engines/table-engines/log-family/log.md +++ b/docs/zh/engines/table-engines/log-family/log.md @@ -1,5 +1,5 @@ -# 日志 {#log} +# Log {#log} -日志与 TinyLog 的不同之处在于，«标记» 的小文件与列文件存在一起。这些标记写在每个数据块上，并且包含偏移量，这些偏移量指示从哪里开始读取文件以便跳过指定的行数。这使得可以在多个线程中读取表数据。对于并发数据访问，可以同时执行读取操作，而写入操作则阻塞读取和其它写入。Log 引擎不支持索引。同样，如果写入表失败，则该表将被破坏，并且从该表读取将返回错误。Log 引擎适用于临时数据，write-once 表以及测试或演示目的。 +`Log` 与 `TinyLog` 的不同之处在于，«标记» 的小文件与列文件存在一起。这些标记写在每个数据块上，并且包含偏移量，这些偏移量指示从哪里开始读取文件以便跳过指定的行数。这使得可以在多个线程中读取表数据。对于并发数据访问，可以同时执行读取操作，而写入操作则阻塞读取和其它写入。`Log`引擎不支持索引。同样，如果写入表失败，则该表将被破坏，并且从该表读取将返回错误。`Log`引擎适用于临时数据，write-once 表以及测试或演示目的。 [原始文章](https://clickhouse.tech/docs/zh/operations/table_engines/log/) diff --git a/docs/zh/operations/tips.md b/docs/zh/operations/tips.md index 4a8cf2efb96..511e8a22644 100644 --- a/docs/zh/operations/tips.md +++ b/docs/zh/operations/tips.md @@ -11,9 +11,9 @@ 不要禁用超线程。 它有助于某些查询，但不适用于其他查询。 -## 涡轮增压 {#turbo-boost} +## 超频 {#turbo-boost} -强烈推荐涡轮增压。 它显着提高了典型负载的性能。 +强烈推荐超频(turbo-boost)。 它显着提高了典型负载的性能。 您可以使用 `turbostat` 要查看负载下的CPU的实际时钟速率。 ## CPU缩放调控器 {#cpu-scaling-governor} @@ -39,18 +39,18 @@ echo 'performance' | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_gover 始终禁用交换文件。 不这样做的唯一原因是，如果您使用的ClickHouse在您的个人笔记本电脑。 -## 巨大的页面 {#huge-pages} +## 大页(Huge Pages) {#huge-pages} -始终禁用透明巨大的页面。 它会干扰内存分alloc，从而导致显着的性能下降。 +始终禁用透明大页(transparent huge pages)。 它会干扰内存分alloc，从而导致显着的性能下降。 ``` bash echo 'never' | sudo tee /sys/kernel/mm/transparent_hugepage/enabled ``` -使用 `perf top` 观看内核中用于内存管理的时间。 -永久巨大的页面也不需要被分配。 +使用 `perf top` 观察内核中用于内存管理的时间。 +永久大页(permanent huge pages)也不需要被分配。 -## 存储子系统 {#storage-subsystem} +## 存储系统 {#storage-subsystem} 如果您的预算允许您使用SSD，请使用SSD。 如果没有，请使用硬盘。 SATA硬盘7200转就行了。 @@ -100,27 +100,27 @@ XFS也是合适的，但它还没有经过ClickHouse的彻底测试。 如果可能的话，至少使用一个10GB的网络。 1Gb也可以工作，但对于使用数十tb的数据修补副本或处理具有大量中间数据的分布式查询，情况会更糟。 -## 动物园管理员 {#zookeeper} +## Zookeeper {#zookeeper} 您可能已经将ZooKeeper用于其他目的。 您可以使用相同的zookeeper安装，如果它还没有超载。 -It’s best to use a fresh version of ZooKeeper – 3.4.9 or later. The version in stable Linux distributions may be outdated. +最好使用新版本的 Zookeeper – 3.4.9 或之后的版本. 稳定 Liunx 发行版中的 Zookeeper 版本可能是落后的。 -You should never use manually written scripts to transfer data between different ZooKeeper clusters, because the result will be incorrect for sequential nodes. Never use the «zkcopy» utility for the same reason: https://github.com/ksprojects/zkcopy/issues/15 +你永远不该使用自己手写的脚本在不同的 Zookeeper 集群之间转移数据, 这可能会导致序列节点的数据不正确。出于同样的原因，永远不要使用 zkcopy 工具: https://github.com/ksprojects/zkcopy/issues/15 如果要将现有ZooKeeper集群分为两个，正确的方法是增加其副本的数量，然后将其重新配置为两个独立的集群。 -不要在与ClickHouse相同的服务器上运行ZooKeeper。 由于ZooKeeper对延迟非常敏感，ClickHouse可能会利用所有可用的系统资源。 +不要在与ClickHouse相同的服务器上运行ZooKeeper。 因为ZooKeeper对延迟非常敏感，而ClickHouse可能会占用所有可用的系统资源。 -使用默认设置，ZooKeeper是一个定时炸弹: +默认设置下，ZooKeeper 就像是一个定时炸弹: -> 使用默认配置时，ZooKeeper服务器不会从旧快照和日志中删除文件（请参阅autopurge），这是操作员的责任。 +当使用默认配置时，ZooKeeper服务不会从旧快照和日志中删除文件（请参阅autopurge），这是操作员的责任。 -必须拆除炸弹 +必须拆除炸弹。 -下面的ZooKeeper（3.5.1）配置在Yandex中使用。梅地卡生产环境截至2017年5月20日: +下面的ZooKeeper（3.5.1）配置在 Yandex.Metrica 的生产环境中使用截至2017年5月20日: -动物园cfg: +zoo.cfg: ``` bash # http://hadoop.apache.org/zookeeper/docs/current/zookeeperAdmin.html @@ -222,7 +222,7 @@ JAVA_OPTS="-Xms{{ '{{' }} cluster.get('xms','128M') {{ '}}' }} \ -XX:+CMSParallelRemarkEnabled" ``` -盐初始化: +Salt init: description "zookeeper-{{ '{{' }} cluster['name'] {{ '}}' }} centralized coordination service" diff --git a/programs/client/Client.cpp b/programs/client/Client.cpp index fc452c165e7..c7884b813cf 100644 --- a/programs/client/Client.cpp +++ b/programs/client/Client.cpp @@ -54,6 +54,7 @@ #include #include #include +#include #include #include #include @@ -1158,13 +1159,13 @@ private: ASTPtr ast_to_process; try { - std::stringstream dump_before_fuzz; + WriteBufferFromOwnString dump_before_fuzz; fuzz_base->dumpTree(dump_before_fuzz); auto base_before_fuzz = fuzz_base->formatForErrorMessage(); ast_to_process = fuzz_base->clone(); - std::stringstream dump_of_cloned_ast; + WriteBufferFromOwnString dump_of_cloned_ast; ast_to_process->dumpTree(dump_of_cloned_ast); // Run the original query as well. @@ -1186,7 +1187,9 @@ private: fprintf(stderr, "dump of cloned ast:\n%s\n", dump_of_cloned_ast.str().c_str()); fprintf(stderr, "dump after fuzz:\n"); - fuzz_base->dumpTree(std::cerr); + WriteBufferFromOStream cerr_buf(std::cerr, 4096); + fuzz_base->dumpTree(cerr_buf); + cerr_buf.next(); fmt::print(stderr, "IAST::clone() is broken for some AST node. This is a bug. The original AST ('dump before fuzz') and its cloned copy ('dump of cloned AST') refer to the same nodes, which must never happen. This means that their parent node doesn't implement clone() correctly."); @@ -1529,7 +1532,9 @@ private: if (is_interactive) { std::cout << std::endl; - formatAST(*res, std::cout); + WriteBufferFromOStream res_buf(std::cout, 4096); + formatAST(*res, res_buf); + res_buf.next(); std::cout << std::endl << std::endl; } diff --git a/programs/client/QueryFuzzer.cpp b/programs/client/QueryFuzzer.cpp index 3fe6b8087f0..a8e32d87db5 100644 --- a/programs/client/QueryFuzzer.cpp +++ b/programs/client/QueryFuzzer.cpp @@ -8,6 +8,7 @@ #include #include #include +#include #include #include #include @@ -419,7 +420,9 @@ void QueryFuzzer::fuzzMain(ASTPtr & ast) fuzz(ast); std::cout << std::endl; - formatAST(*ast, std::cout, false /*highlight*/); + WriteBufferFromOStream ast_buf(std::cout, 4096); + formatAST(*ast, ast_buf, false /*highlight*/); + ast_buf.next(); std::cout << std::endl << std::endl; } diff --git a/programs/client/Suggest.cpp b/programs/client/Suggest.cpp index e85e7a21261..87083c2c27b 100644 --- a/programs/client/Suggest.cpp +++ b/programs/client/Suggest.cpp @@ -86,7 +86,7 @@ Suggest::Suggest() void Suggest::loadImpl(Connection & connection, const ConnectionTimeouts & timeouts, size_t suggestion_limit) { - std::stringstream query; + std::stringstream query; // STYLE_CHECK_ALLOW_STD_STRING_STREAM query << "SELECT DISTINCT arrayJoin(extractAll(name, '[\\\\w_]{2,}')) AS res FROM (" "SELECT name FROM system.functions" " UNION ALL " diff --git a/programs/client/TestHint.h b/programs/client/TestHint.h index 641c3e0ccf0..65666f4304c 100644 --- a/programs/client/TestHint.h +++ b/programs/client/TestHint.h @@ -93,7 +93,7 @@ private: void parse(const String & hint) { - std::stringstream ss; + std::stringstream ss; // STYLE_CHECK_ALLOW_STD_STRING_STREAM ss << hint; String item; diff --git a/programs/copier/ClusterCopier.cpp b/programs/copier/ClusterCopier.cpp index 4ee14b14119..a129dc7efcc 100644 --- a/programs/copier/ClusterCopier.cpp +++ b/programs/copier/ClusterCopier.cpp @@ -162,7 +162,7 @@ void ClusterCopier::discoverShardPartitions(const ConnectionTimeouts & timeouts, if (!missing_partitions.empty()) { - std::stringstream ss; + WriteBufferFromOwnString ss; for (const String & missing_partition : missing_partitions) ss << " " << missing_partition; diff --git a/programs/copier/Internals.cpp b/programs/copier/Internals.cpp index 0f607ea5faf..ea2be469945 100644 --- a/programs/copier/Internals.cpp +++ b/programs/copier/Internals.cpp @@ -13,7 +13,7 @@ using ConfigurationPtr = Poco::AutoPtr; ConfigurationPtr getConfigurationFromXMLString(const std::string & xml_data) { - std::stringstream ss(xml_data); + std::stringstream ss(xml_data); // STYLE_CHECK_ALLOW_STD_STRING_STREAM Poco::XML::InputSource input_source{ss}; return {new Poco::Util::XMLConfiguration{&input_source}}; } diff --git a/programs/copier/TaskTableAndShard.h b/programs/copier/TaskTableAndShard.h index cc12fe556c7..4f5bfb443e6 100644 --- a/programs/copier/TaskTableAndShard.h +++ b/programs/copier/TaskTableAndShard.h @@ -394,12 +394,8 @@ inline ASTPtr TaskTable::rewriteReplicatedCreateQueryToPlain() inline String DB::TaskShard::getDescription() const { - std::stringstream ss; - ss << "N" << numberInCluster() - << " (having a replica " << getHostNameExample() - << ", pull table " + getQuotedTable(task_table.table_pull) - << " of cluster " + task_table.cluster_pull_name << ")"; - return ss.str(); + return fmt::format("N{} (having a replica {}, pull table {} of cluster {}", + numberInCluster(), getHostNameExample(), getQuotedTable(task_table.table_pull), task_table.cluster_pull_name); } inline String DB::TaskShard::getHostNameExample() const diff --git a/programs/format/Format.cpp b/programs/format/Format.cpp index 01f952bf95e..986dc67a798 100644 --- a/programs/format/Format.cpp +++ b/programs/format/Format.cpp @@ -6,6 +6,7 @@ #include #include #include +#include #include #include #include @@ -129,7 +130,9 @@ int mainEntryClickHouseFormat(int argc, char ** argv) ASTPtr res = parseQueryAndMovePosition(parser, pos, end, "query", multiple, 0, DBMS_DEFAULT_MAX_PARSER_DEPTH); if (!quiet) { - formatAST(*res, std::cout, hilite, oneline); + WriteBufferFromOStream res_buf(std::cout, 4096); + formatAST(*res, res_buf, hilite, oneline); + res_buf.next(); if (multiple) std::cout << "\n;\n"; std::cout << std::endl; diff --git a/programs/local/LocalServer.cpp b/programs/local/LocalServer.cpp index 9ecc2a50a42..15e71198eb1 100644 --- a/programs/local/LocalServer.cpp +++ b/programs/local/LocalServer.cpp @@ -422,7 +422,7 @@ static const char * minimal_default_user_xml = static ConfigurationPtr getConfigurationFromXMLString(const char * xml_data) { - std::stringstream ss{std::string{xml_data}}; + std::stringstream ss{std::string{xml_data}}; // STYLE_CHECK_ALLOW_STD_STRING_STREAM Poco::XML::InputSource input_source{ss}; return {new Poco::Util::XMLConfiguration{&input_source}}; } diff --git a/programs/odbc-bridge/ColumnInfoHandler.cpp b/programs/odbc-bridge/ColumnInfoHandler.cpp index 33723b4cc8e..9e2307715c5 100644 --- a/programs/odbc-bridge/ColumnInfoHandler.cpp +++ b/programs/odbc-bridge/ColumnInfoHandler.cpp @@ -113,16 +113,16 @@ void ODBCColumnsInfoHandler::handleRequest(Poco::Net::HTTPServerRequest & reques /// TODO Why not do SQLColumns instead? std::string name = schema_name.empty() ? backQuoteIfNeed(table_name) : backQuoteIfNeed(schema_name) + "." + backQuoteIfNeed(table_name); - std::stringstream ss; + WriteBufferFromOwnString buf; std::string input = "SELECT * FROM " + name + " WHERE 1 = 0"; ParserQueryWithOutput parser; ASTPtr select = parseQuery(parser, input.data(), input.data() + input.size(), "", context_settings.max_query_size, context_settings.max_parser_depth); - IAST::FormatSettings settings(ss, true); + IAST::FormatSettings settings(buf, true); settings.always_quote_identifiers = true; settings.identifier_quoting_style = getQuotingStyle(hdbc); select->format(settings); - std::string query = ss.str(); + std::string query = buf.str(); LOG_TRACE(log, "Inferring structure with query '{}'", query); diff --git a/programs/odbc-bridge/ODBCBlockOutputStream.cpp b/programs/odbc-bridge/ODBCBlockOutputStream.cpp index 82ca861ea67..4d8b9fa6bdf 100644 --- a/programs/odbc-bridge/ODBCBlockOutputStream.cpp +++ b/programs/odbc-bridge/ODBCBlockOutputStream.cpp @@ -32,12 +32,12 @@ namespace for (const auto & column : columns) query.columns->children.emplace_back(std::make_shared(column.name)); - std::stringstream ss; - IAST::FormatSettings settings(ss, true); + WriteBufferFromOwnString buf; + IAST::FormatSettings settings(buf, true); settings.always_quote_identifiers = true; settings.identifier_quoting_style = quoting; query.IAST::format(settings); - return ss.str(); + return buf.str(); } std::string getQuestionMarks(size_t n) diff --git a/programs/server/Server.cpp b/programs/server/Server.cpp index ed18793a537..951ece89929 100644 --- a/programs/server/Server.cpp +++ b/programs/server/Server.cpp @@ -191,10 +191,10 @@ int Server::run() if (config().hasOption("help")) { Poco::Util::HelpFormatter help_formatter(Server::options()); - std::stringstream header; - header << commandName() << " [OPTION] [-- [ARG]...]\n"; - header << "positional arguments can be used to rewrite config.xml properties, for example, --http_port=8010"; - help_formatter.setHeader(header.str()); + auto header_str = fmt::format("{} [OPTION] [-- [ARG]...]\n" + "positional arguments can be used to rewrite config.xml properties, for example, --http_port=8010", + commandName()); + help_formatter.setHeader(header_str); help_formatter.format(std::cout); return 0; } @@ -568,6 +568,8 @@ int Server::main(const std::vector & /*args*/) if (config->has("zookeeper")) global_context->reloadZooKeeperIfChanged(config); + global_context->reloadAuxiliaryZooKeepersConfigIfChanged(config); + global_context->updateStorageConfiguration(*config); }, /* already_loaded = */ true); diff --git a/src/Access/DiskAccessStorage.cpp b/src/Access/DiskAccessStorage.cpp index abf4ff12d5a..426c27ea799 100644 --- a/src/Access/DiskAccessStorage.cpp +++ b/src/Access/DiskAccessStorage.cpp @@ -197,11 +197,13 @@ namespace boost::range::push_back(queries, InterpreterShowGrantsQuery::getAttachGrantQueries(entity)); /// Serialize the list of ATTACH queries to a string. - std::stringstream ss; - ss.exceptions(std::ios::failbit); + WriteBufferFromOwnString buf; for (const ASTPtr & query : queries) - ss << *query << ";\n"; - String file_contents = std::move(ss).str(); + { + formatAST(*query, buf, false, true); + buf.write(";\n", 2); + } + String file_contents = buf.str(); /// First we save *.tmp file and then we rename if everything's ok. auto tmp_file_path = std::filesystem::path{file_path}.replace_extension(".tmp"); @@ -353,7 +355,7 @@ String DiskAccessStorage::getStorageParamsJSON() const json.set("path", directory_path); if (readonly) json.set("readonly", readonly.load()); - std::ostringstream oss; + std::ostringstream oss; // STYLE_CHECK_ALLOW_STD_STRING_STREAM oss.exceptions(std::ios::failbit); Poco::JSON::Stringifier::stringify(json, oss); return oss.str(); diff --git a/src/Access/LDAPAccessStorage.cpp b/src/Access/LDAPAccessStorage.cpp index c9f00f2f4ab..92de7fce8d7 100644 --- a/src/Access/LDAPAccessStorage.cpp +++ b/src/Access/LDAPAccessStorage.cpp @@ -150,7 +150,7 @@ String LDAPAccessStorage::getStorageParamsJSON() const params_json.set("server", ldap_server); params_json.set("roles", default_role_names); - std::ostringstream oss; + std::ostringstream oss; // STYLE_CHECK_ALLOW_STD_STRING_STREAM oss.exceptions(std::ios::failbit); Poco::JSON::Stringifier::stringify(params_json, oss); diff --git a/src/Access/UsersConfigAccessStorage.cpp b/src/Access/UsersConfigAccessStorage.cpp index ba3c2d5b8e5..eb993d696c6 100644 --- a/src/Access/UsersConfigAccessStorage.cpp +++ b/src/Access/UsersConfigAccessStorage.cpp @@ -460,7 +460,7 @@ String UsersConfigAccessStorage::getStorageParamsJSON() const Poco::JSON::Object json; if (!path.empty()) json.set("path", path); - std::ostringstream oss; + std::ostringstream oss; // STYLE_CHECK_ALLOW_STD_STRING_STREAM oss.exceptions(std::ios::failbit); Poco::JSON::Stringifier::stringify(json, oss); return oss.str(); diff --git a/src/AggregateFunctions/AggregateFunctionGroupArray.h b/src/AggregateFunctions/AggregateFunctionGroupArray.h index ba9b235de07..3255ea42edb 100644 --- a/src/AggregateFunctions/AggregateFunctionGroupArray.h +++ b/src/AggregateFunctions/AggregateFunctionGroupArray.h @@ -2,6 +2,9 @@ #include #include +#include +#include +#include #include #include @@ -244,10 +247,9 @@ public: if constexpr (Trait::sampler == Sampler::RNG) { DB::writeIntBinary(this->data(place).total_values, buf); - std::ostringstream rng_stream; - rng_stream.exceptions(std::ios::failbit); - rng_stream << this->data(place).rng; - DB::writeStringBinary(rng_stream.str(), buf); + WriteBufferFromOwnString rng_buf; + rng_buf << this->data(place).rng; + DB::writeStringBinary(rng_buf.str(), buf); } // TODO @@ -275,9 +277,8 @@ public: DB::readIntBinary(this->data(place).total_values, buf); std::string rng_string; DB::readStringBinary(rng_string, buf); - std::istringstream rng_stream(rng_string); - rng_stream.exceptions(std::ios::failbit); - rng_stream >> this->data(place).rng; + ReadBufferFromString rng_buf(rng_string); + rng_buf >> this->data(place).rng; } // TODO @@ -565,10 +566,9 @@ public: if constexpr (Trait::sampler == Sampler::RNG) { DB::writeIntBinary(data(place).total_values, buf); - std::ostringstream rng_stream; - rng_stream.exceptions(std::ios::failbit); - rng_stream << data(place).rng; - DB::writeStringBinary(rng_stream.str(), buf); + WriteBufferFromOwnString rng_buf; + rng_buf << data(place).rng; + DB::writeStringBinary(rng_buf.str(), buf); } // TODO @@ -600,9 +600,8 @@ public: DB::readIntBinary(data(place).total_values, buf); std::string rng_string; DB::readStringBinary(rng_string, buf); - std::istringstream rng_stream(rng_string); - rng_stream.exceptions(std::ios::failbit); - rng_stream >> data(place).rng; + ReadBufferFromString rng_buf(rng_string); + rng_buf >> data(place).rng; } // TODO diff --git a/src/AggregateFunctions/AggregateFunctionRetention.h b/src/AggregateFunctions/AggregateFunctionRetention.h index b742dcdf77f..f8a2163ccb9 100644 --- a/src/AggregateFunctions/AggregateFunctionRetention.h +++ b/src/AggregateFunctions/AggregateFunctionRetention.h @@ -1,7 +1,5 @@ #pragma once -#include -#include #include #include #include diff --git a/src/AggregateFunctions/AggregateFunctionTimeSeriesGroupSum.h b/src/AggregateFunctions/AggregateFunctionTimeSeriesGroupSum.h index b755fbf081b..28c3f53d879 100644 --- a/src/AggregateFunctions/AggregateFunctionTimeSeriesGroupSum.h +++ b/src/AggregateFunctions/AggregateFunctionTimeSeriesGroupSum.h @@ -1,10 +1,8 @@ #pragma once #include -#include #include #include -#include #include #include #include diff --git a/src/AggregateFunctions/AggregateFunctionWindowFunnel.h b/src/AggregateFunctions/AggregateFunctionWindowFunnel.h index 3297819a9ff..2dd6ef8a9fd 100644 --- a/src/AggregateFunctions/AggregateFunctionWindowFunnel.h +++ b/src/AggregateFunctions/AggregateFunctionWindowFunnel.h @@ -1,7 +1,5 @@ #pragma once -#include -#include #include #include #include diff --git a/src/AggregateFunctions/QuantileExact.h b/src/AggregateFunctions/QuantileExact.h index de909c27565..a2acde97f90 100644 --- a/src/AggregateFunctions/QuantileExact.h +++ b/src/AggregateFunctions/QuantileExact.h @@ -8,7 +8,9 @@ #include #include -#include +#if !defined(ARCADIA_BUILD) + #include // Y_IGNORE +#endif namespace DB { @@ -88,7 +90,11 @@ struct QuantileExact : QuantileExactBase> { size_t n = level < 1 ? level * array.size() : (array.size() - 1); +#if !defined(ARCADIA_BUILD) miniselect::floyd_rivest_select(array.begin(), array.begin() + n, array.end()); /// NOTE You can think of the radix-select algorithm. +#else + std::nth_element(array.begin(), array.begin() + n, array.end()); /// NOTE You can think of the radix-select algorithm. +#endif return array[n]; } @@ -108,8 +114,11 @@ struct QuantileExact : QuantileExactBase> size_t n = level < 1 ? level * array.size() : (array.size() - 1); +#if !defined(ARCADIA_BUILD) miniselect::floyd_rivest_select(array.begin() + prev_n, array.begin() + n, array.end()); - +#else + std::nth_element(array.begin() + prev_n, array.begin() + n, array.end()); +#endif result[indices[i]] = array[n]; prev_n = n; } @@ -145,7 +154,11 @@ struct QuantileExactExclusive : public QuantileExact else if (n < 1) return static_cast(array[0]); +#if !defined(ARCADIA_BUILD) miniselect::floyd_rivest_select(array.begin(), array.begin() + n - 1, array.end()); +#else + std::nth_element(array.begin(), array.begin() + n - 1, array.end()); +#endif auto nth_element = std::min_element(array.begin() + n, array.end()); return static_cast(array[n - 1]) + (h - n) * static_cast(*nth_element - array[n - 1]); @@ -174,7 +187,11 @@ struct QuantileExactExclusive : public QuantileExact result[indices[i]] = static_cast(array[0]); else { +#if !defined(ARCADIA_BUILD) miniselect::floyd_rivest_select(array.begin() + prev_n, array.begin() + n - 1, array.end()); +#else + std::nth_element(array.begin() + prev_n, array.begin() + n - 1, array.end()); +#endif auto nth_element = std::min_element(array.begin() + n, array.end()); result[indices[i]] = static_cast(array[n - 1]) + (h - n) * static_cast(*nth_element - array[n - 1]); @@ -209,8 +226,11 @@ struct QuantileExactInclusive : public QuantileExact return static_cast(array[array.size() - 1]); else if (n < 1) return static_cast(array[0]); - +#if !defined(ARCADIA_BUILD) miniselect::floyd_rivest_select(array.begin(), array.begin() + n - 1, array.end()); +#else + std::nth_element(array.begin(), array.begin() + n - 1, array.end()); +#endif auto nth_element = std::min_element(array.begin() + n, array.end()); return static_cast(array[n - 1]) + (h - n) * static_cast(*nth_element - array[n - 1]); @@ -237,7 +257,11 @@ struct QuantileExactInclusive : public QuantileExact result[indices[i]] = static_cast(array[0]); else { +#if !defined(ARCADIA_BUILD) miniselect::floyd_rivest_select(array.begin() + prev_n, array.begin() + n - 1, array.end()); +#else + std::nth_element(array.begin() + prev_n, array.begin() + n - 1, array.end()); +#endif auto nth_element = std::min_element(array.begin() + n, array.end()); result[indices[i]] = static_cast(array[n - 1]) + (h - n) * static_cast(*nth_element - array[n - 1]); diff --git a/src/AggregateFunctions/QuantileTiming.h b/src/AggregateFunctions/QuantileTiming.h index 28bcde5c140..1a696088dd4 100644 --- a/src/AggregateFunctions/QuantileTiming.h +++ b/src/AggregateFunctions/QuantileTiming.h @@ -7,7 +7,9 @@ #include #include -#include +#if !defined(ARCADIA_BUILD) + #include // Y_IGNORE +#endif namespace DB { @@ -180,7 +182,11 @@ namespace detail /// Sorting an array will not be considered a violation of constancy. auto & array = elems; +#if !defined(ARCADIA_BUILD) miniselect::floyd_rivest_select(array.begin(), array.begin() + n, array.end()); +#else + std::nth_element(array.begin(), array.begin() + n, array.end()); +#endif quantile = array[n]; } @@ -201,7 +207,11 @@ namespace detail ? level * elems.size() : (elems.size() - 1); +#if !defined(ARCADIA_BUILD) miniselect::floyd_rivest_select(array.begin() + prev_n, array.begin() + n, array.end()); +#else + std::nth_element(array.begin() + prev_n, array.begin() + n, array.end()); +#endif result[level_index] = array[n]; prev_n = n; diff --git a/src/AggregateFunctions/ReservoirSampler.h b/src/AggregateFunctions/ReservoirSampler.h index f82b0b856a0..a321ef08648 100644 --- a/src/AggregateFunctions/ReservoirSampler.h +++ b/src/AggregateFunctions/ReservoirSampler.h @@ -3,11 +3,13 @@ #include #include #include -#include #include #include #include #include +#include +#include +#include #include #include #include @@ -190,9 +192,8 @@ public: std::string rng_string; DB::readStringBinary(rng_string, buf); - std::istringstream rng_stream(rng_string); - rng_stream.exceptions(std::ios::failbit); - rng_stream >> rng; + DB::ReadBufferFromString rng_buf(rng_string); + rng_buf >> rng; for (size_t i = 0; i < samples.size(); ++i) DB::readBinary(samples[i], buf); @@ -205,10 +206,9 @@ public: DB::writeIntBinary(sample_count, buf); DB::writeIntBinary(total_values, buf); - std::ostringstream rng_stream; - rng_stream.exceptions(std::ios::failbit); - rng_stream << rng; - DB::writeStringBinary(rng_stream.str(), buf); + DB::WriteBufferFromOwnString rng_buf; + rng_buf << rng; + DB::writeStringBinary(rng_buf.str(), buf); for (size_t i = 0; i < std::min(sample_count, total_values); ++i) DB::writeBinary(samples[i], buf); diff --git a/src/AggregateFunctions/ReservoirSamplerDeterministic.h b/src/AggregateFunctions/ReservoirSamplerDeterministic.h index eae24c1f3e9..3b7817e9308 100644 --- a/src/AggregateFunctions/ReservoirSamplerDeterministic.h +++ b/src/AggregateFunctions/ReservoirSamplerDeterministic.h @@ -3,7 +3,6 @@ #include #include #include -#include #include #include #include diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index 2ac990acfff..9fc8a62a066 100644 --- a/src/CMakeLists.txt +++ b/src/CMakeLists.txt @@ -335,6 +335,13 @@ if (ZSTD_LIBRARY) endif () endif() +set (LZMA_LIBRARY liblzma) +set (LZMA_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/contrib/xz/src/liblzma/api) +if (LZMA_LIBRARY) + target_link_libraries (clickhouse_common_io PUBLIC ${LZMA_LIBRARY}) + target_include_directories (clickhouse_common_io SYSTEM BEFORE PUBLIC ${LZMA_INCLUDE_DIR}) +endif() + if (USE_ICU) dbms_target_link_libraries (PRIVATE ${ICU_LIBRARIES}) dbms_target_include_directories (SYSTEM PRIVATE ${ICU_INCLUDE_DIRS}) diff --git a/src/Client/MultiplexedConnections.cpp b/src/Client/MultiplexedConnections.cpp index b6cb55fa979..a99b0f9d7cc 100644 --- a/src/Client/MultiplexedConnections.cpp +++ b/src/Client/MultiplexedConnections.cpp @@ -1,5 +1,6 @@ #include #include +#include #include @@ -222,19 +223,18 @@ std::string MultiplexedConnections::dumpAddresses() const std::string MultiplexedConnections::dumpAddressesUnlocked() const { bool is_first = true; - std::ostringstream os; - os.exceptions(std::ios::failbit); + WriteBufferFromOwnString buf; for (const ReplicaState & state : replica_states) { const Connection * connection = state.connection; if (connection) { - os << (is_first ? "" : "; ") << connection->getDescription(); + buf << (is_first ? "" : "; ") << connection->getDescription(); is_first = false; } } - return os.str(); + return buf.str(); } Packet MultiplexedConnections::receivePacketUnlocked() diff --git a/src/Columns/ColumnArray.cpp b/src/Columns/ColumnArray.cpp index 6dbe755f0ba..82d954df334 100644 --- a/src/Columns/ColumnArray.cpp +++ b/src/Columns/ColumnArray.cpp @@ -20,7 +20,10 @@ #include #include -#include +#if !defined(ARCADIA_BUILD) + #include // Y_IGNORE +#endif + namespace DB { @@ -783,7 +786,11 @@ void ColumnArray::getPermutationImpl(size_t limit, Permutation & res, Comparator auto less = [&cmp](size_t lhs, size_t rhs){ return cmp(lhs, rhs) < 0; }; if (limit) +#if !defined(ARCADIA_BUILD) miniselect::floyd_rivest_partial_sort(res.begin(), res.begin() + limit, res.end(), less); +#else + std::partial_sort(res.begin(), res.begin() + limit, res.end(), less); +#endif else std::sort(res.begin(), res.end(), less); } @@ -835,8 +842,11 @@ void ColumnArray::updatePermutationImpl(size_t limit, Permutation & res, EqualRa return; /// Since then we are working inside the interval. - +#if !defined(ARCADIA_BUILD) miniselect::floyd_rivest_partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, less); +#else + std::partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, less); +#endif auto new_first = first; for (auto j = first + 1; j < limit; ++j) { diff --git a/src/Columns/ColumnDecimal.cpp b/src/Columns/ColumnDecimal.cpp index 4285259a4f4..7c3af5fe095 100644 --- a/src/Columns/ColumnDecimal.cpp +++ b/src/Columns/ColumnDecimal.cpp @@ -8,7 +8,10 @@ #include #include -#include +#if !defined(ARCADIA_BUILD) + #include // Y_IGNORE +#endif + #include @@ -194,12 +197,21 @@ void ColumnDecimal::updatePermutation(bool reverse, size_t limit, int, IColum /// Since then we are working inside the interval. if (reverse) +#if !defined(ARCADIA_BUILD) miniselect::floyd_rivest_partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, [this](size_t a, size_t b) { return data[a] > data[b]; }); +#else + std::partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, + [this](size_t a, size_t b) { return data[a] > data[b]; }); +#endif else +#if !defined(ARCADIA_BUILD) miniselect::floyd_rivest_partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, [this](size_t a, size_t b) { return data[a] < data[b]; }); - +#else + std::partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, + [this](size_t a, size_t b) { return data[a] > data[b]; }); +#endif auto new_first = first; for (auto j = first + 1; j < limit; ++j) { diff --git a/src/Columns/ColumnDecimal.h b/src/Columns/ColumnDecimal.h index 128b595803f..abb49531e89 100644 --- a/src/Columns/ColumnDecimal.h +++ b/src/Columns/ColumnDecimal.h @@ -7,7 +7,9 @@ #include #include #include -#include +#if !defined(ARCADIA_BUILD) + #include // Y_IGNORE +#endif namespace DB @@ -254,9 +256,17 @@ protected: sort_end = res.begin() + limit; if (reverse) +#if !defined(ARCADIA_BUILD) miniselect::floyd_rivest_partial_sort(res.begin(), sort_end, res.end(), [this](size_t a, size_t b) { return data[a] > data[b]; }); +#else + std::partial_sort(res.begin(), sort_end, res.end(), [this](size_t a, size_t b) { return data[a] > data[b]; }); +#endif else +#if !defined(ARCADIA_BUILD) miniselect::floyd_rivest_partial_sort(res.begin(), sort_end, res.end(), [this](size_t a, size_t b) { return data[a] < data[b]; }); +#else + std::partial_sort(res.begin(), sort_end, res.end(), [this](size_t a, size_t b) { return data[a] < data[b]; }); +#endif } }; diff --git a/src/Columns/ColumnFixedString.cpp b/src/Columns/ColumnFixedString.cpp index 41e46a7fa98..c1724b37fe2 100644 --- a/src/Columns/ColumnFixedString.cpp +++ b/src/Columns/ColumnFixedString.cpp @@ -10,7 +10,9 @@ #include #include -#include +#if !defined(ARCADIA_BUILD) + #include // Y_IGNORE +#endif #include @@ -158,9 +160,17 @@ void ColumnFixedString::getPermutation(bool reverse, size_t limit, int /*nan_dir if (limit) { if (reverse) +#if !defined(ARCADIA_BUILD) miniselect::floyd_rivest_partial_sort(res.begin(), res.begin() + limit, res.end(), less(*this)); +#else + std::partial_sort(res.begin(), res.begin() + limit, res.end(), less(*this)); +#endif else +#if !defined(ARCADIA_BUILD) miniselect::floyd_rivest_partial_sort(res.begin(), res.begin() + limit, res.end(), less(*this)); +#else + std::partial_sort(res.begin(), res.begin() + limit, res.end(), less(*this)); +#endif } else { @@ -218,9 +228,17 @@ void ColumnFixedString::updatePermutation(bool reverse, size_t limit, int, Permu /// Since then we are working inside the interval. if (reverse) +#if !defined(ARCADIA_BUILD) miniselect::floyd_rivest_partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, less(*this)); +#else + std::partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, less(*this)); +#endif else +#if !defined(ARCADIA_BUILD) miniselect::floyd_rivest_partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, less(*this)); +#else + std::partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, less(*this)); +#endif auto new_first = first; for (auto j = first + 1; j < limit; ++j) diff --git a/src/Columns/ColumnLowCardinality.cpp b/src/Columns/ColumnLowCardinality.cpp index 9b87a409aaa..f61062160b1 100644 --- a/src/Columns/ColumnLowCardinality.cpp +++ b/src/Columns/ColumnLowCardinality.cpp @@ -8,7 +8,10 @@ #include #include -#include + +#if !defined(ARCADIA_BUILD) + #include // Y_IGNORE +#endif namespace DB { @@ -394,7 +397,11 @@ void ColumnLowCardinality::updatePermutationImpl(size_t limit, Permutation & res /// Since then we are working inside the interval. +#if !defined(ARCADIA_BUILD) miniselect::floyd_rivest_partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, less); +#else + std::partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, less); +#endif auto new_first = first; for (auto j = first + 1; j < limit; ++j) diff --git a/src/Columns/ColumnString.cpp b/src/Columns/ColumnString.cpp index 541863486a6..49180919abb 100644 --- a/src/Columns/ColumnString.cpp +++ b/src/Columns/ColumnString.cpp @@ -10,7 +10,10 @@ #include #include -#include +#if !defined(ARCADIA_BUILD) + #include // Y_IGNORE +#endif + namespace DB { @@ -314,7 +317,11 @@ void ColumnString::getPermutationImpl(size_t limit, Permutation & res, Comparato auto less = [&cmp](size_t lhs, size_t rhs){ return cmp(lhs, rhs) < 0; }; if (limit) +#if !defined(ARCADIA_BUILD) miniselect::floyd_rivest_partial_sort(res.begin(), res.begin() + limit, res.end(), less); +#else + std::partial_sort(res.begin(), res.begin() + limit, res.end(), less); +#endif else std::sort(res.begin(), res.end(), less); } @@ -365,8 +372,11 @@ void ColumnString::updatePermutationImpl(size_t limit, Permutation & res, EqualR return; /// Since then we are working inside the interval. - +#if !defined(ARCADIA_BUILD) miniselect::floyd_rivest_partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, less); +#else + std::partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, less); +#endif size_t new_first = first; for (size_t j = first + 1; j < limit; ++j) diff --git a/src/Columns/ColumnTuple.cpp b/src/Columns/ColumnTuple.cpp index cbe5c7e11cd..9130f563735 100644 --- a/src/Columns/ColumnTuple.cpp +++ b/src/Columns/ColumnTuple.cpp @@ -9,7 +9,10 @@ #include #include #include -#include +#if !defined(ARCADIA_BUILD) + #include // Y_IGNORE +#endif + namespace DB { @@ -352,7 +355,11 @@ void ColumnTuple::getPermutationImpl(size_t limit, Permutation & res, LessOperat if (limit) { +#if !defined(ARCADIA_BUILD) miniselect::floyd_rivest_partial_sort(res.begin(), res.begin() + limit, res.end(), less); +#else + std::partial_sort(res.begin(), res.begin() + limit, res.end(), less); +#endif } else { diff --git a/src/Columns/ColumnVector.cpp b/src/Columns/ColumnVector.cpp index e9af38d6984..c02963e4c5a 100644 --- a/src/Columns/ColumnVector.cpp +++ b/src/Columns/ColumnVector.cpp @@ -17,7 +17,9 @@ #include #include #include -#include +#if !defined(ARCADIA_BUILD) + #include // Y_IGNORE +#endif #ifdef __SSE2__ #include @@ -156,9 +158,17 @@ void ColumnVector::getPermutation(bool reverse, size_t limit, int nan_directi res[i] = i; if (reverse) +#if !defined(ARCADIA_BUILD) miniselect::floyd_rivest_partial_sort(res.begin(), res.begin() + limit, res.end(), greater(*this, nan_direction_hint)); +#else + std::partial_sort(res.begin(), res.begin() + limit, res.end(), greater(*this, nan_direction_hint)); +#endif else +#if !defined(ARCADIA_BUILD) miniselect::floyd_rivest_partial_sort(res.begin(), res.begin() + limit, res.end(), less(*this, nan_direction_hint)); +#else + std::partial_sort(res.begin(), res.begin() + limit, res.end(), less(*this, nan_direction_hint)); +#endif } else { @@ -254,9 +264,17 @@ void ColumnVector::updatePermutation(bool reverse, size_t limit, int nan_dire /// Since then, we are working inside the interval. if (reverse) +#if !defined(ARCADIA_BUILD) miniselect::floyd_rivest_partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, greater(*this, nan_direction_hint)); +#else + std::partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, greater(*this, nan_direction_hint)); +#endif else +#if !defined(ARCADIA_BUILD) miniselect::floyd_rivest_partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, less(*this, nan_direction_hint)); +#else + std::partial_sort(res.begin() + first, res.begin() + limit, res.begin() + last, less(*this, nan_direction_hint)); +#endif size_t new_first = first; for (size_t j = first + 1; j < limit; ++j) diff --git a/src/Columns/tests/gtest_weak_hash_32.cpp b/src/Columns/tests/gtest_weak_hash_32.cpp index a7fd60a3b9c..a04bd94124c 100644 --- a/src/Columns/tests/gtest_weak_hash_32.cpp +++ b/src/Columns/tests/gtest_weak_hash_32.cpp @@ -71,7 +71,7 @@ void checkColumn( std::unordered_map map; size_t num_collisions = 0; - std::stringstream collisions_str; + std::stringstream collisions_str; // STYLE_CHECK_ALLOW_STD_STRING_STREAM collisions_str.exceptions(std::ios::failbit); for (size_t i = 0; i < eq_class.size(); ++i) diff --git a/src/Common/Config/ConfigProcessor.cpp b/src/Common/Config/ConfigProcessor.cpp index 8a6093c47c9..3e06be94b11 100644 --- a/src/Common/Config/ConfigProcessor.cpp +++ b/src/Common/Config/ConfigProcessor.cpp @@ -5,7 +5,6 @@ #include #include #include -#include #include #include #include @@ -17,6 +16,8 @@ #include #include #include +#include +#include #define PREPROCESSED_SUFFIX "-preprocessed" @@ -537,8 +538,7 @@ XMLDocumentPtr ConfigProcessor::processConfig( if (has_zk_includes) *has_zk_includes = !contributing_zk_paths.empty(); - std::stringstream comment; - comment.exceptions(std::ios::failbit); + WriteBufferFromOwnString comment; comment << " This file was generated automatically.\n"; comment << " Do not edit it: it is likely to be discarded and generated again before it's read next time.\n"; comment << " Files used to generate this file:"; diff --git a/src/Common/ErrorCodes.cpp b/src/Common/ErrorCodes.cpp index 74daa4ebbed..405b8c60af8 100644 --- a/src/Common/ErrorCodes.cpp +++ b/src/Common/ErrorCodes.cpp @@ -520,34 +520,34 @@ M(551, ILLEGAL_PROJECTION_MANIPULATOR) \ M(552, UNRECOGNIZED_ARGUMENTS) \ M(553, ROCKSDB_ERROR) \ - \ + M(553, LZMA_STREAM_ENCODER_FAILED) \ + M(554, LZMA_STREAM_DECODER_FAILED) \ M(999, KEEPER_EXCEPTION) \ M(1000, POCO_EXCEPTION) \ M(1001, STD_EXCEPTION) \ - M(1002, UNKNOWN_EXCEPTION) \ + M(1002, UNKNOWN_EXCEPTION) /* See END */ namespace DB { - namespace ErrorCodes { - #define M(VALUE, NAME) extern const Value NAME = VALUE; - APPLY_FOR_ERROR_CODES(M) - #undef M +#define M(VALUE, NAME) extern const Value NAME = VALUE; + APPLY_FOR_ERROR_CODES(M) +#undef M constexpr Value END = 3000; - std::atomic values[END + 1] {}; + std::atomic values[END + 1]{}; struct ErrorCodesNames { std::string_view names[END + 1]; ErrorCodesNames() { - #define M(VALUE, NAME) names[VALUE] = std::string_view(#NAME); - APPLY_FOR_ERROR_CODES(M) - #undef M +#define M(VALUE, NAME) names[VALUE] = std::string_view(#NAME); + APPLY_FOR_ERROR_CODES(M) +#undef M } } error_codes_names; @@ -558,7 +558,7 @@ namespace ErrorCodes return error_codes_names.names[error_code]; } - ErrorCode end() { return END+1; } + ErrorCode end() { return END + 1; } } } diff --git a/src/Common/Exception.cpp b/src/Common/Exception.cpp index 820e3857bfc..dd78d0ec9fc 100644 --- a/src/Common/Exception.cpp +++ b/src/Common/Exception.cpp @@ -245,8 +245,7 @@ static std::string getExtraExceptionInfo(const std::exception & e) std::string getCurrentExceptionMessage(bool with_stacktrace, bool check_embedded_stacktrace /*= false*/, bool with_extra_info /*= true*/) { - std::stringstream stream; - stream.exceptions(std::ios::failbit); + WriteBufferFromOwnString stream; try { @@ -365,8 +364,7 @@ void tryLogException(std::exception_ptr e, Poco::Logger * logger, const std::str std::string getExceptionMessage(const Exception & e, bool with_stacktrace, bool check_embedded_stacktrace) { - std::stringstream stream; - stream.exceptions(std::ios::failbit); + WriteBufferFromOwnString stream; try { diff --git a/src/Common/HTMLForm.h b/src/Common/HTMLForm.h index 2490d613160..2b62167dce7 100644 --- a/src/Common/HTMLForm.h +++ b/src/Common/HTMLForm.h @@ -16,13 +16,13 @@ struct HTMLForm : public Poco::Net::HTMLForm HTMLForm(const Poco::Net::HTTPRequest & request) { Poco::URI uri(request.getURI()); - std::istringstream istr(uri.getRawQuery()); + std::istringstream istr(uri.getRawQuery()); // STYLE_CHECK_ALLOW_STD_STRING_STREAM readUrl(istr); } HTMLForm(const Poco::URI & uri) { - std::istringstream istr(uri.getRawQuery()); + std::istringstream istr(uri.getRawQuery()); // STYLE_CHECK_ALLOW_STD_STRING_STREAM readUrl(istr); } diff --git a/src/Common/MemoryTracker.cpp b/src/Common/MemoryTracker.cpp index 5257f95898a..9f6db025886 100644 --- a/src/Common/MemoryTracker.cpp +++ b/src/Common/MemoryTracker.cpp @@ -133,17 +133,13 @@ void MemoryTracker::alloc(Int64 size) BlockerInThread untrack_lock; ProfileEvents::increment(ProfileEvents::QueryMemoryLimitExceeded); - std::stringstream message; - message.exceptions(std::ios::failbit); - message << "Memory tracker"; - if (const auto * description = description_ptr.load(std::memory_order_relaxed)) - message << " " << description; - message << ": fault injected. Would use " << formatReadableSizeWithBinarySuffix(will_be) - << " (attempt to allocate chunk of " << size << " bytes)" - << ", maximum: " << formatReadableSizeWithBinarySuffix(current_hard_limit); - + const auto * description = description_ptr.load(std::memory_order_relaxed); amount.fetch_sub(size, std::memory_order_relaxed); - throw DB::Exception(message.str(), DB::ErrorCodes::MEMORY_LIMIT_EXCEEDED); + throw DB::Exception(DB::ErrorCodes::MEMORY_LIMIT_EXCEEDED, + "Memory tracker{}{}: fault injected. Would use {} (attempt to allocate chunk of {} bytes), maximum: {}", + description ? " " : "", description ? description : "", + formatReadableSizeWithBinarySuffix(will_be), + size, formatReadableSizeWithBinarySuffix(current_hard_limit)); } if (unlikely(current_profiler_limit && will_be > current_profiler_limit)) @@ -166,17 +162,13 @@ void MemoryTracker::alloc(Int64 size) BlockerInThread untrack_lock; ProfileEvents::increment(ProfileEvents::QueryMemoryLimitExceeded); - std::stringstream message; - message.exceptions(std::ios::failbit); - message << "Memory limit"; - if (const auto * description = description_ptr.load(std::memory_order_relaxed)) - message << " " << description; - message << " exceeded: would use " << formatReadableSizeWithBinarySuffix(will_be) - << " (attempt to allocate chunk of " << size << " bytes)" - << ", maximum: " << formatReadableSizeWithBinarySuffix(current_hard_limit); - + const auto * description = description_ptr.load(std::memory_order_relaxed); amount.fetch_sub(size, std::memory_order_relaxed); - throw DB::Exception(message.str(), DB::ErrorCodes::MEMORY_LIMIT_EXCEEDED); + throw DB::Exception(DB::ErrorCodes::MEMORY_LIMIT_EXCEEDED, + "Memory limit{}{} exceeded: would use {} (attempt to allocate chunk of {} bytes), maximum: {}", + description ? " " : "", description ? description : "", + formatReadableSizeWithBinarySuffix(will_be), + size, formatReadableSizeWithBinarySuffix(current_hard_limit)); } updatePeak(will_be); diff --git a/src/Common/ShellCommand.cpp b/src/Common/ShellCommand.cpp index db0928ea605..069c7774729 100644 --- a/src/Common/ShellCommand.cpp +++ b/src/Common/ShellCommand.cpp @@ -8,6 +8,7 @@ #include #include #include +#include #include #include @@ -73,8 +74,7 @@ ShellCommand::~ShellCommand() void ShellCommand::logCommand(const char * filename, char * const argv[]) { - std::stringstream args; - args.exceptions(std::ios::failbit); + WriteBufferFromOwnString args; for (int i = 0; argv != nullptr && argv[i] != nullptr; ++i) { if (i > 0) diff --git a/src/Common/StackTrace.cpp b/src/Common/StackTrace.cpp index 7e9474ac3a7..9cfdba2687f 100644 --- a/src/Common/StackTrace.cpp +++ b/src/Common/StackTrace.cpp @@ -23,7 +23,7 @@ std::string signalToErrorMessage(int sig, const siginfo_t & info, const ucontext_t & context) { - std::stringstream error; + std::stringstream error; // STYLE_CHECK_ALLOW_STD_STRING_STREAM error.exceptions(std::ios::failbit); switch (sig) { @@ -319,7 +319,7 @@ static void toStringEveryLineImpl( const DB::SymbolIndex & symbol_index = DB::SymbolIndex::instance(); std::unordered_map dwarfs; - std::stringstream out; + std::stringstream out; // STYLE_CHECK_ALLOW_STD_STRING_STREAM out.exceptions(std::ios::failbit); for (size_t i = offset; i < size; ++i) @@ -359,7 +359,7 @@ static void toStringEveryLineImpl( out.str({}); } #else - std::stringstream out; + std::stringstream out; // STYLE_CHECK_ALLOW_STD_STRING_STREAM out.exceptions(std::ios::failbit); for (size_t i = offset; i < size; ++i) @@ -375,7 +375,7 @@ static void toStringEveryLineImpl( static std::string toStringImpl(const StackTrace::FramePointers & frame_pointers, size_t offset, size_t size) { - std::stringstream out; + std::stringstream out; // STYLE_CHECK_ALLOW_STD_STRING_STREAM out.exceptions(std::ios::failbit); toStringEveryLineImpl(frame_pointers, offset, size, [&](const std::string & str) { out << str << '\n'; }); return out.str(); diff --git a/src/Common/StudentTTest.cpp b/src/Common/StudentTTest.cpp index fe605092acc..da4913126b3 100644 --- a/src/Common/StudentTTest.cpp +++ b/src/Common/StudentTTest.cpp @@ -153,7 +153,7 @@ std::pair StudentTTest::compareAndReport(size_t confidence_le double mean_confidence_interval = table_value * t_statistic; - std::stringstream ss; + std::stringstream ss; // STYLE_CHECK_ALLOW_STD_STRING_STREAM ss.exceptions(std::ios::failbit); if (mean_difference > mean_confidence_interval && (mean_difference - mean_confidence_interval > 0.0001)) /// difference must be more than 0.0001, to take into account connection latency. diff --git a/src/Common/ThreadProfileEvents.cpp b/src/Common/ThreadProfileEvents.cpp index 7b94ca0f2b2..c8a4555dabf 100644 --- a/src/Common/ThreadProfileEvents.cpp +++ b/src/Common/ThreadProfileEvents.cpp @@ -413,7 +413,8 @@ std::vector PerfEventsCounters::eventIndicesFromString(const std::string return result; } - std::istringstream iss(events_list); + + std::istringstream iss(events_list); // STYLE_CHECK_ALLOW_STD_STRING_STREAM std::string event_name; while (std::getline(iss, event_name, ',')) { diff --git a/src/Common/ThreadStatus.cpp b/src/Common/ThreadStatus.cpp index f5ad28f57af..dad32293380 100644 --- a/src/Common/ThreadStatus.cpp +++ b/src/Common/ThreadStatus.cpp @@ -1,5 +1,3 @@ -#include - #include #include #include @@ -79,12 +77,10 @@ void ThreadStatus::assertState(const std::initializer_list & permitted_stat return; } - std::stringstream ss; - ss.exceptions(std::ios::failbit); - ss << "Unexpected thread state " << getCurrentState(); if (description) - ss << ": " << description; - throw Exception(ss.str(), ErrorCodes::LOGICAL_ERROR); + throw Exception(ErrorCodes::LOGICAL_ERROR, "Unexpected thread state {}: {}", getCurrentState(), description); + else + throw Exception(ErrorCodes::LOGICAL_ERROR, "Unexpected thread state {}", getCurrentState()); } void ThreadStatus::attachInternalTextLogsQueue(const InternalTextLogsQueuePtr & logs_queue, diff --git a/src/Common/UInt128.h b/src/Common/UInt128.h index 7b6f8e7c7be..baa33578089 100644 --- a/src/Common/UInt128.h +++ b/src/Common/UInt128.h @@ -1,11 +1,11 @@ #pragma once #include -#include #include #include #include +#include #ifdef __SSE4_2__ #include @@ -48,10 +48,9 @@ struct UInt128 String toHexString() const { - std::ostringstream os; - os.exceptions(std::ios::failbit); - os << std::setw(16) << std::setfill('0') << std::hex << high << low; - return String(os.str()); + String res(2 * sizeof(UInt128), 0); + writeHexUIntLowercase(*this, res.data()); + return res; } bool inline operator== (const UInt128 rhs) const { return tuple() == rhs.tuple(); } diff --git a/src/Common/XDBCBridgeHelper.h b/src/Common/XDBCBridgeHelper.h index c794d2fe3cd..59bae33d88d 100644 --- a/src/Common/XDBCBridgeHelper.h +++ b/src/Common/XDBCBridgeHelper.h @@ -1,6 +1,5 @@ #pragma once -#include #include #include #include @@ -307,9 +306,6 @@ struct ODBCBridgeMixin std::vector cmd_args; path.setFileName("clickhouse-odbc-bridge"); - std::stringstream command; - command.exceptions(std::ios::failbit); - #if !CLICKHOUSE_SPLIT_BINARY cmd_args.push_back("odbc-bridge"); #endif diff --git a/src/Common/ZooKeeper/TestKeeper.cpp b/src/Common/ZooKeeper/TestKeeper.cpp index f7db8a85f96..4ae26d874fb 100644 --- a/src/Common/ZooKeeper/TestKeeper.cpp +++ b/src/Common/ZooKeeper/TestKeeper.cpp @@ -218,7 +218,7 @@ std::pair TestKeeperCreateRequest::process(TestKeeper::Contai auto seq_num = it->second.seq_num; ++it->second.seq_num; - std::stringstream seq_num_str; + std::stringstream seq_num_str; // STYLE_CHECK_ALLOW_STD_STRING_STREAM seq_num_str.exceptions(std::ios::failbit); seq_num_str << std::setw(10) << std::setfill('0') << seq_num; diff --git a/src/Common/checkStackSize.cpp b/src/Common/checkStackSize.cpp index bdc117eccac..e94abc08c6b 100644 --- a/src/Common/checkStackSize.cpp +++ b/src/Common/checkStackSize.cpp @@ -3,7 +3,6 @@ #include #include #include -#include #if defined(__FreeBSD__) # include @@ -80,13 +79,8 @@ __attribute__((__weak__)) void checkStackSize() /// It's safe to assume that overflow in multiplying by two cannot occur. if (stack_size * 2 > max_stack_size) { - std::stringstream message; - message.exceptions(std::ios::failbit); - message << "Stack size too large" - << ". Stack address: " << stack_address - << ", frame address: " << frame_address - << ", stack size: " << stack_size - << ", maximum stack size: " << max_stack_size; - throw Exception(message.str(), ErrorCodes::TOO_DEEP_RECURSION); + throw Exception(ErrorCodes::TOO_DEEP_RECURSION, + "Stack size too large. Stack address: {}, frame address: {}, stack size: {}, maximum stack size: {}", + stack_address, frame_address, stack_size, max_stack_size); } } diff --git a/src/Common/formatReadable.cpp b/src/Common/formatReadable.cpp index 99929275521..fc5c6c19b50 100644 --- a/src/Common/formatReadable.cpp +++ b/src/Common/formatReadable.cpp @@ -1,6 +1,4 @@ #include -#include -#include #include #include diff --git a/src/Common/parseGlobs.cpp b/src/Common/parseGlobs.cpp index f04ad1ea8a0..75539512b6d 100644 --- a/src/Common/parseGlobs.cpp +++ b/src/Common/parseGlobs.cpp @@ -1,4 +1,7 @@ #include +#include +#include +#include #include #include #include @@ -18,21 +21,21 @@ namespace DB */ std::string makeRegexpPatternFromGlobs(const std::string & initial_str_with_globs) { - std::ostringstream oss_for_escaping; - oss_for_escaping.exceptions(std::ios::failbit); + /// FIXME make it better + WriteBufferFromOwnString buf_for_escaping; /// Escaping only characters that not used in glob syntax for (const auto & letter : initial_str_with_globs) { - if ((letter == '[') || (letter == ']') || (letter == '|') || (letter == '+') || (letter == '-') || (letter == '(') || (letter == ')')) - oss_for_escaping << '\\'; - oss_for_escaping << letter; + if ((letter == '[') || (letter == ']') || (letter == '|') || (letter == '+') || (letter == '-') || (letter == '(') || (letter == ')') || (letter == '\\')) + buf_for_escaping << '\\'; + buf_for_escaping << letter; } - std::string escaped_with_globs = oss_for_escaping.str(); + std::string escaped_with_globs = buf_for_escaping.str(); static const re2::RE2 enum_or_range(R"({([\d]+\.\.[\d]+|[^{}*,]+,[^{}*]*[^{}*,])})"); /// regexp for {expr1,expr2,expr3} or {M..N}, where M and N - non-negative integers, expr's should be without {}*, re2::StringPiece input(escaped_with_globs); re2::StringPiece matched; - std::ostringstream oss_for_replacing; + std::ostringstream oss_for_replacing; // STYLE_CHECK_ALLOW_STD_STRING_STREAM oss_for_replacing.exceptions(std::ios::failbit); size_t current_index = 0; while (RE2::FindAndConsume(&input, enum_or_range, &matched)) @@ -45,9 +48,8 @@ std::string makeRegexpPatternFromGlobs(const std::string & initial_str_with_glob size_t range_begin = 0; size_t range_end = 0; char point; - std::istringstream iss_range(buffer); - iss_range.exceptions(std::ios::failbit); - iss_range >> range_begin >> point >> point >> range_end; + ReadBufferFromString buf_range(buffer); + buf_range >> range_begin >> point >> point >> range_end; bool leading_zeros = buffer[0] == '0'; size_t num_len = std::to_string(range_end).size(); if (leading_zeros) @@ -71,20 +73,19 @@ std::string makeRegexpPatternFromGlobs(const std::string & initial_str_with_glob } oss_for_replacing << escaped_with_globs.substr(current_index); std::string almost_res = oss_for_replacing.str(); - std::ostringstream oss_final_processing; - oss_final_processing.exceptions(std::ios::failbit); + WriteBufferFromOwnString buf_final_processing; for (const auto & letter : almost_res) { if ((letter == '?') || (letter == '*')) { - oss_final_processing << "[^/]"; /// '?' is any symbol except '/' + buf_final_processing << "[^/]"; /// '?' is any symbol except '/' if (letter == '?') continue; } if ((letter == '.') || (letter == '{') || (letter == '}')) - oss_final_processing << '\\'; - oss_final_processing << letter; + buf_final_processing << '\\'; + buf_final_processing << letter; } - return oss_final_processing.str(); + return buf_final_processing.str(); } } diff --git a/src/Common/tests/gtest_getMultipleValuesFromConfig.cpp b/src/Common/tests/gtest_getMultipleValuesFromConfig.cpp index 4756043acbf..30aba9418ba 100644 --- a/src/Common/tests/gtest_getMultipleValuesFromConfig.cpp +++ b/src/Common/tests/gtest_getMultipleValuesFromConfig.cpp @@ -9,7 +9,8 @@ using namespace DB; TEST(Common, getMultipleValuesFromConfig) { - std::istringstream xml_isteam(R"END( + std::istringstream // STYLE_CHECK_ALLOW_STD_STRING_STREAM + xml_isteam(R"END( 0 diff --git a/src/Common/tests/gtest_sensitive_data_masker.cpp b/src/Common/tests/gtest_sensitive_data_masker.cpp index 67ad5be2f52..b8125c77b9b 100644 --- a/src/Common/tests/gtest_sensitive_data_masker.cpp +++ b/src/Common/tests/gtest_sensitive_data_masker.cpp @@ -102,7 +102,8 @@ TEST(Common, SensitiveDataMasker) EXPECT_EQ(maskerbad.wipeSensitiveData(x), 0); { - std::istringstream xml_isteam(R"END( + std::istringstream // STYLE_CHECK_ALLOW_STD_STRING_STREAM + xml_isteam(R"END( @@ -152,7 +153,8 @@ TEST(Common, SensitiveDataMasker) try { - std::istringstream xml_isteam_bad(R"END( + std::istringstream // STYLE_CHECK_ALLOW_STD_STRING_STREAM + xml_isteam_bad(R"END( @@ -181,7 +183,8 @@ TEST(Common, SensitiveDataMasker) try { - std::istringstream xml_isteam_bad(R"END( + std::istringstream // STYLE_CHECK_ALLOW_STD_STRING_STREAM + xml_isteam_bad(R"END( test @@ -203,7 +206,8 @@ TEST(Common, SensitiveDataMasker) try { - std::istringstream xml_isteam_bad(R"END( + std::istringstream // STYLE_CHECK_ALLOW_STD_STRING_STREAM + xml_isteam_bad(R"END( test())( diff --git a/src/Compression/CompressedReadBufferBase.cpp b/src/Compression/CompressedReadBufferBase.cpp index 7a6b605d015..8b16b68a999 100644 --- a/src/Compression/CompressedReadBufferBase.cpp +++ b/src/Compression/CompressedReadBufferBase.cpp @@ -12,6 +12,7 @@ #include #include #include +#include namespace ProfileEvents @@ -42,7 +43,7 @@ static void validateChecksum(char * data, size_t size, const Checksum expected_c if (expected_checksum == calculated_checksum) return; - std::stringstream message; + WriteBufferFromOwnString message; /// TODO mess up of endianness in error message. message << "Checksum doesn't match: corrupted data." @@ -50,7 +51,16 @@ static void validateChecksum(char * data, size_t size, const Checksum expected_c + ". Actual: " + getHexUIntLowercase(calculated_checksum.first) + getHexUIntLowercase(calculated_checksum.second) + ". Size of compressed block: " + toString(size); - const char * message_hardware_failure = "This is most likely due to hardware failure. If you receive broken data over network and the error does not repeat every time, this can be caused by bad RAM on network interface controller or bad controller itself or bad RAM on network switches or bad CPU on network switches (look at the logs on related network switches; note that TCP checksums don't help) or bad RAM on host (look at dmesg or kern.log for enormous amount of EDAC errors, ECC-related reports, Machine Check Exceptions, mcelog; note that ECC memory can fail if the number of errors is huge) or bad CPU on host. If you read data from disk, this can be caused by disk bit rott. This exception protects ClickHouse from data corruption due to hardware failures."; + const char * message_hardware_failure = "This is most likely due to hardware failure. " + "If you receive broken data over network and the error does not repeat every time, " + "this can be caused by bad RAM on network interface controller or bad controller itself " + "or bad RAM on network switches or bad CPU on network switches " + "(look at the logs on related network switches; note that TCP checksums don't help) " + "or bad RAM on host (look at dmesg or kern.log for enormous amount of EDAC errors, " + "ECC-related reports, Machine Check Exceptions, mcelog; note that ECC memory can fail " + "if the number of errors is huge) or bad CPU on host. If you read data from disk, " + "this can be caused by disk bit rott. This exception protects ClickHouse " + "from data corruption due to hardware failures."; auto flip_bit = [](char * buf, size_t pos) { diff --git a/src/Compression/tests/compressed_buffer.cpp b/src/Compression/tests/compressed_buffer.cpp index c018fc95995..aef2cf4ab90 100644 --- a/src/Compression/tests/compressed_buffer.cpp +++ b/src/Compression/tests/compressed_buffer.cpp @@ -51,10 +51,7 @@ int main(int, char **) if (x != i) { - std::stringstream s; - s.exceptions(std::ios::failbit); - s << "Failed!, read: " << x << ", expected: " << i; - throw DB::Exception(s.str(), 0); + throw DB::Exception(0, "Failed!, read: {}, expected: {}", x, i); } } stopwatch.stop(); diff --git a/src/Core/MySQL/IMySQLReadPacket.cpp b/src/Core/MySQL/IMySQLReadPacket.cpp index 676f3986ba4..74f309d0294 100644 --- a/src/Core/MySQL/IMySQLReadPacket.cpp +++ b/src/Core/MySQL/IMySQLReadPacket.cpp @@ -1,5 +1,4 @@ #include -#include #include #include @@ -21,10 +20,9 @@ void IMySQLReadPacket::readPayload(ReadBuffer & in, uint8_t & sequence_id) readPayloadImpl(payload); if (!payload.eof()) { - std::stringstream tmp; - tmp.exceptions(std::ios::failbit); - tmp << "Packet payload is not fully read. Stopped after " << payload.count() << " bytes, while " << payload.available() << " bytes are in buffer."; - throw Exception(tmp.str(), ErrorCodes::UNKNOWN_PACKET_FROM_CLIENT); + throw Exception(ErrorCodes::UNKNOWN_PACKET_FROM_CLIENT, + "Packet payload is not fully read. Stopped after {} bytes, while {} bytes are in buffer.", + payload.count(), payload.available()); } } diff --git a/src/Core/MySQL/IMySQLWritePacket.cpp b/src/Core/MySQL/IMySQLWritePacket.cpp index 3e97800177c..b5c95717a9b 100644 --- a/src/Core/MySQL/IMySQLWritePacket.cpp +++ b/src/Core/MySQL/IMySQLWritePacket.cpp @@ -1,10 +1,14 @@ #include #include -#include namespace DB { +namespace ErrorCodes +{ + extern const int LOGICAL_ERROR; +} + namespace MySQLProtocol { @@ -15,10 +19,8 @@ void IMySQLWritePacket::writePayload(WriteBuffer & buffer, uint8_t & sequence_id buf.next(); if (buf.remainingPayloadSize()) { - std::stringstream ss; - ss.exceptions(std::ios::failbit); - ss << "Incomplete payload. Written " << getPayloadSize() - buf.remainingPayloadSize() << " bytes, expected " << getPayloadSize() << " bytes."; - throw Exception(ss.str(), 0); + throw Exception(ErrorCodes::LOGICAL_ERROR, "Incomplete payload. Written {} bytes, expected {} bytes.", + getPayloadSize() - buf.remainingPayloadSize(), getPayloadSize()); } } diff --git a/src/Core/MySQL/MySQLReplication.cpp b/src/Core/MySQL/MySQLReplication.cpp index c09c4b3b034..6ff1670777a 100644 --- a/src/Core/MySQL/MySQLReplication.cpp +++ b/src/Core/MySQL/MySQLReplication.cpp @@ -4,6 +4,7 @@ #include #include #include +#include #include #include #include @@ -35,15 +36,15 @@ namespace MySQLReplication payload.readStrict(reinterpret_cast(&flags), 2); } - void EventHeader::dump(std::ostream & out) const + void EventHeader::dump(WriteBuffer & out) const { - out << "\n=== " << to_string(this->type) << " ===" << std::endl; - out << "Timestamp: " << this->timestamp << std::endl; - out << "Event Type: " << this->type << std::endl; - out << "Server ID: " << this->server_id << std::endl; - out << "Event Size: " << this->event_size << std::endl; - out << "Log Pos: " << this->log_pos << std::endl; - out << "Flags: " << this->flags << std::endl; + out << "\n=== " << to_string(this->type) << " ===" << '\n'; + out << "Timestamp: " << this->timestamp << '\n'; + out << "Event Type: " << to_string(this->type) << '\n'; + out << "Server ID: " << this->server_id << '\n'; + out << "Event Size: " << this->event_size << '\n'; + out << "Log Pos: " << this->log_pos << '\n'; + out << "Flags: " << this->flags << '\n'; } /// https://dev.mysql.com/doc/internals/en/format-description-event.html @@ -60,13 +61,13 @@ namespace MySQLReplication readStringUntilEOF(event_type_header_length, payload); } - void FormatDescriptionEvent::dump(std::ostream & out) const + void FormatDescriptionEvent::dump(WriteBuffer & out) const { header.dump(out); - out << "Binlog Version: " << this->binlog_version << std::endl; - out << "Server Version: " << this->server_version << std::endl; - out << "Create Timestamp: " << this->create_timestamp << std::endl; - out << "Event Header Len: " << std::to_string(this->event_header_length) << std::endl; + out << "Binlog Version: " << this->binlog_version << '\n'; + out << "Server Version: " << this->server_version << '\n'; + out << "Create Timestamp: " << this->create_timestamp << '\n'; + out << "Event Header Len: " << std::to_string(this->event_header_length) << '\n'; } /// https://dev.mysql.com/doc/internals/en/rotate-event.html @@ -76,11 +77,11 @@ namespace MySQLReplication readStringUntilEOF(next_binlog, payload); } - void RotateEvent::dump(std::ostream & out) const + void RotateEvent::dump(WriteBuffer & out) const { header.dump(out); - out << "Position: " << this->position << std::endl; - out << "Next Binlog: " << this->next_binlog << std::endl; + out << "Position: " << this->position << '\n'; + out << "Next Binlog: " << this->next_binlog << '\n'; } /// https://dev.mysql.com/doc/internals/en/query-event.html @@ -116,24 +117,24 @@ namespace MySQLReplication } } - void QueryEvent::dump(std::ostream & out) const + void QueryEvent::dump(WriteBuffer & out) const { header.dump(out); - out << "Thread ID: " << this->thread_id << std::endl; - out << "Execution Time: " << this->exec_time << std::endl; - out << "Schema Len: " << std::to_string(this->schema_len) << std::endl; - out << "Error Code: " << this->error_code << std::endl; - out << "Status Len: " << this->status_len << std::endl; - out << "Schema: " << this->schema << std::endl; - out << "Query: " << this->query << std::endl; + out << "Thread ID: " << this->thread_id << '\n'; + out << "Execution Time: " << this->exec_time << '\n'; + out << "Schema Len: " << std::to_string(this->schema_len) << '\n'; + out << "Error Code: " << this->error_code << '\n'; + out << "Status Len: " << this->status_len << '\n'; + out << "Schema: " << this->schema << '\n'; + out << "Query: " << this->query << '\n'; } void XIDEvent::parseImpl(ReadBuffer & payload) { payload.readStrict(reinterpret_cast(&xid), 8); } - void XIDEvent::dump(std::ostream & out) const + void XIDEvent::dump(WriteBuffer & out) const { header.dump(out); - out << "XID: " << this->xid << std::endl; + out << "XID: " << this->xid << '\n'; } void TableMapEvent::parseImpl(ReadBuffer & payload) @@ -238,21 +239,23 @@ namespace MySQLReplication } } - void TableMapEvent::dump(std::ostream & out) const + void TableMapEvent::dump(WriteBuffer & out) const { header.dump(out); - out << "Table ID: " << this->table_id << std::endl; - out << "Flags: " << this->flags << std::endl; - out << "Schema Len: " << std::to_string(this->schema_len) << std::endl; - out << "Schema: " << this->schema << std::endl; - out << "Table Len: " << std::to_string(this->table_len) << std::endl; - out << "Table: " << this->table << std::endl; - out << "Column Count: " << this->column_count << std::endl; + out << "Table ID: " << this->table_id << '\n'; + out << "Flags: " << this->flags << '\n'; + out << "Schema Len: " << std::to_string(this->schema_len) << '\n'; + out << "Schema: " << this->schema << '\n'; + out << "Table Len: " << std::to_string(this->table_len) << '\n'; + out << "Table: " << this->table << '\n'; + out << "Column Count: " << this->column_count << '\n'; for (auto i = 0U; i < column_count; i++) { - out << "Column Type [" << i << "]: " << std::to_string(column_type[i]) << ", Meta: " << column_meta[i] << std::endl; + out << "Column Type [" << i << "]: " << std::to_string(column_type[i]) << ", Meta: " << column_meta[i] << '\n'; } - out << "Null Bitmap: " << this->null_bitmap << std::endl; + String bitmap_str; + boost::to_string(this->null_bitmap, bitmap_str); + out << "Null Bitmap: " << bitmap_str << '\n'; } void RowsEvent::parseImpl(ReadBuffer & payload) @@ -631,16 +634,16 @@ namespace MySQLReplication rows.push_back(row); } - void RowsEvent::dump(std::ostream & out) const + void RowsEvent::dump(WriteBuffer & out) const { FieldVisitorToString to_string; header.dump(out); - out << "Schema: " << this->schema << std::endl; - out << "Table: " << this->table << std::endl; + out << "Schema: " << this->schema << '\n'; + out << "Table: " << this->table << '\n'; for (auto i = 0U; i < rows.size(); i++) { - out << "Row[" << i << "]: " << applyVisitor(to_string, rows[i]) << std::endl; + out << "Row[" << i << "]: " << applyVisitor(to_string, rows[i]) << '\n'; } } @@ -663,22 +666,22 @@ namespace MySQLReplication payload.ignoreAll(); } - void GTIDEvent::dump(std::ostream & out) const + void GTIDEvent::dump(WriteBuffer & out) const { WriteBufferFromOwnString ws; writeUUIDText(gtid.uuid, ws); auto gtid_next = ws.str() + ":" + std::to_string(gtid.seq_no); header.dump(out); - out << "GTID Next: " << gtid_next << std::endl; + out << "GTID Next: " << gtid_next << '\n'; } void DryRunEvent::parseImpl(ReadBuffer & payload) { payload.ignoreAll(); } - void DryRunEvent::dump(std::ostream & out) const + void DryRunEvent::dump(WriteBuffer & out) const { header.dump(out); - out << "[DryRun Event]" << std::endl; + out << "[DryRun Event]" << '\n'; } /// Update binlog name/position/gtid based on the event type. @@ -716,12 +719,12 @@ namespace MySQLReplication gtid_sets.parse(gtid_sets_); } - void Position::dump(std::ostream & out) const + void Position::dump(WriteBuffer & out) const { - out << "\n=== Binlog Position ===" << std::endl; - out << "Binlog: " << this->binlog_name << std::endl; - out << "Position: " << this->binlog_pos << std::endl; - out << "GTIDSets: " << this->gtid_sets.toString() << std::endl; + out << "\n=== Binlog Position ===" << '\n'; + out << "Binlog: " << this->binlog_name << '\n'; + out << "Position: " << this->binlog_pos << '\n'; + out << "GTIDSets: " << this->gtid_sets.toString() << '\n'; } void MySQLFlavor::readPayloadImpl(ReadBuffer & payload) diff --git a/src/Core/MySQL/MySQLReplication.h b/src/Core/MySQL/MySQLReplication.h index 6f5b4cf0a1e..394ac729d1b 100644 --- a/src/Core/MySQL/MySQLReplication.h +++ b/src/Core/MySQL/MySQLReplication.h @@ -309,7 +309,7 @@ namespace MySQLReplication UInt16 flags; EventHeader() : timestamp(0), server_id(0), event_size(0), log_pos(0), flags(0) { } - void dump(std::ostream & out) const; + void dump(WriteBuffer & out) const; void parse(ReadBuffer & payload); }; @@ -321,7 +321,7 @@ namespace MySQLReplication EventBase(EventHeader && header_) : header(std::move(header_)) {} virtual ~EventBase() = default; - virtual void dump(std::ostream & out) const = 0; + virtual void dump(WriteBuffer & out) const = 0; virtual void parseEvent(ReadBuffer & payload) { parseImpl(payload); } virtual MySQLEventType type() const { return MYSQL_UNHANDLED_EVENT; } @@ -344,7 +344,7 @@ namespace MySQLReplication UInt8 event_header_length; String event_type_header_length; - void dump(std::ostream & out) const override; + void dump(WriteBuffer & out) const override; void parseImpl(ReadBuffer & payload) override; private: @@ -358,7 +358,7 @@ namespace MySQLReplication String next_binlog; RotateEvent(EventHeader && header_) : EventBase(std::move(header_)), position(0) {} - void dump(std::ostream & out) const override; + void dump(WriteBuffer & out) const override; protected: void parseImpl(ReadBuffer & payload) override; @@ -389,7 +389,7 @@ namespace MySQLReplication { } - void dump(std::ostream & out) const override; + void dump(WriteBuffer & out) const override; MySQLEventType type() const override { return MYSQL_QUERY_EVENT; } protected: @@ -404,7 +404,7 @@ namespace MySQLReplication protected: UInt64 xid; - void dump(std::ostream & out) const override; + void dump(WriteBuffer & out) const override; void parseImpl(ReadBuffer & payload) override; }; @@ -423,7 +423,7 @@ namespace MySQLReplication Bitmap null_bitmap; TableMapEvent(EventHeader && header_) : EventBase(std::move(header_)), table_id(0), flags(0), schema_len(0), table_len(0), column_count(0) {} - void dump(std::ostream & out) const override; + void dump(WriteBuffer & out) const override; protected: void parseImpl(ReadBuffer & payload) override; @@ -445,7 +445,7 @@ namespace MySQLReplication table = table_map->table; } - void dump(std::ostream & out) const override; + void dump(WriteBuffer & out) const override; protected: UInt64 table_id; @@ -489,7 +489,7 @@ namespace MySQLReplication GTID gtid; GTIDEvent(EventHeader && header_) : EventBase(std::move(header_)), commit_flag(0) {} - void dump(std::ostream & out) const override; + void dump(WriteBuffer & out) const override; protected: void parseImpl(ReadBuffer & payload) override; @@ -499,7 +499,7 @@ namespace MySQLReplication { public: DryRunEvent(EventHeader && header_) : EventBase(std::move(header_)) {} - void dump(std::ostream & out) const override; + void dump(WriteBuffer & out) const override; protected: void parseImpl(ReadBuffer & payload) override; @@ -515,7 +515,7 @@ namespace MySQLReplication Position() : binlog_pos(0) { } void update(BinlogEventPtr event); void update(UInt64 binlog_pos_, const String & binlog_name_, const String & gtid_sets_); - void dump(std::ostream & out) const; + void dump(WriteBuffer & out) const; }; class IFlavor : public MySQLProtocol::IMySQLReadPacket diff --git a/src/Core/SortDescription.h b/src/Core/SortDescription.h index 98229bb73d7..79ac7ddf142 100644 --- a/src/Core/SortDescription.h +++ b/src/Core/SortDescription.h @@ -60,10 +60,7 @@ struct SortColumnDescription std::string dump() const { - std::stringstream ss; - ss.exceptions(std::ios::failbit); - ss << column_name << ":" << column_number << ":dir " << direction << "nulls " << nulls_direction; - return ss.str(); + return fmt::format("{}:{}:dir {}nulls ", column_name, column_number, direction, nulls_direction); } }; diff --git a/src/Core/tests/mysql_protocol.cpp b/src/Core/tests/mysql_protocol.cpp index 7e6aae5da23..9dc46891241 100644 --- a/src/Core/tests/mysql_protocol.cpp +++ b/src/Core/tests/mysql_protocol.cpp @@ -7,6 +7,7 @@ #include #include #include +#include #include @@ -329,6 +330,8 @@ int main(int argc, char ** argv) slave.connect(); slave.startBinlogDumpGTID(slave_id, replicate_db, gtid_sets); + WriteBufferFromOStream cerr(std::cerr); + /// Read one binlog event on by one. while (true) { @@ -337,40 +340,40 @@ int main(int argc, char ** argv) { case MYSQL_QUERY_EVENT: { auto binlog_event = std::static_pointer_cast(event); - binlog_event->dump(std::cerr); + binlog_event->dump(cerr); Position pos = slave.getPosition(); - pos.dump(std::cerr); + pos.dump(cerr); break; } case MYSQL_WRITE_ROWS_EVENT: { auto binlog_event = std::static_pointer_cast(event); - binlog_event->dump(std::cerr); + binlog_event->dump(cerr); Position pos = slave.getPosition(); - pos.dump(std::cerr); + pos.dump(cerr); break; } case MYSQL_UPDATE_ROWS_EVENT: { auto binlog_event = std::static_pointer_cast(event); - binlog_event->dump(std::cerr); + binlog_event->dump(cerr); Position pos = slave.getPosition(); - pos.dump(std::cerr); + pos.dump(cerr); break; } case MYSQL_DELETE_ROWS_EVENT: { auto binlog_event = std::static_pointer_cast(event); - binlog_event->dump(std::cerr); + binlog_event->dump(cerr); Position pos = slave.getPosition(); - pos.dump(std::cerr); + pos.dump(cerr); break; } default: if (event->header.type != MySQLReplication::EventType::HEARTBEAT_EVENT) { - event->dump(std::cerr); + event->dump(cerr); } break; } diff --git a/src/DataStreams/CheckConstraintsBlockOutputStream.cpp b/src/DataStreams/CheckConstraintsBlockOutputStream.cpp index d47d82689de..d4a892711ef 100644 --- a/src/DataStreams/CheckConstraintsBlockOutputStream.cpp +++ b/src/DataStreams/CheckConstraintsBlockOutputStream.cpp @@ -59,15 +59,10 @@ void CheckConstraintsBlockOutputStream::write(const Block & block) /// Is violated. if (!value) { - std::stringstream exception_message; - exception_message.exceptions(std::ios::failbit); - - exception_message << "Constraint " << backQuote(constraint_ptr->name) - << " for table " << table_id.getNameForLogs() - << " is violated, because it is a constant expression returning 0." - << " It is most likely an error in table definition."; - - throw Exception{exception_message.str(), ErrorCodes::VIOLATED_CONSTRAINT}; + throw Exception(ErrorCodes::VIOLATED_CONSTRAINT, + "Constraint {} for table {} is violated, because it is a constant expression returning 0. " + "It is most likely an error in table definition.", + backQuote(constraint_ptr->name), table_id.getNameForLogs()); } } else @@ -87,28 +82,27 @@ void CheckConstraintsBlockOutputStream::write(const Block & block) Names related_columns = constraint_expr->getRequiredColumns(); - std::stringstream exception_message; - exception_message.exceptions(std::ios::failbit); - - exception_message << "Constraint " << backQuote(constraint_ptr->name) - << " for table " << table_id.getNameForLogs() - << " is violated at row " << (rows_written + row_idx + 1) - << ". Expression: (" << serializeAST(*(constraint_ptr->expr), true) << ")" - << ". Column values"; - bool first = true; + String column_values_msg; + constexpr size_t approx_bytes_for_col = 32; + column_values_msg.reserve(approx_bytes_for_col * related_columns.size()); for (const auto & name : related_columns) { const IColumn & column = *block.getByName(name).column; assert(row_idx < column.size()); - exception_message << (first ? ": " : ", ") - << backQuoteIfNeed(name) << " = " << applyVisitor(FieldVisitorToString(), column[row_idx]); - + if (!first) + column_values_msg.append(", "); + column_values_msg.append(backQuoteIfNeed(name)); + column_values_msg.append(" = "); + column_values_msg.append(applyVisitor(FieldVisitorToString(), column[row_idx])); first = false; } - throw Exception{exception_message.str(), ErrorCodes::VIOLATED_CONSTRAINT}; + throw Exception(ErrorCodes::VIOLATED_CONSTRAINT, + "Constraint {} for table {} is violated at row {}. Expression: ({}). Column values: {}", + backQuote(constraint_ptr->name), table_id.getNameForLogs(), rows_written + row_idx + 1, + serializeAST(*(constraint_ptr->expr), true), column_values_msg); } } } diff --git a/src/DataStreams/IBlockInputStream.cpp b/src/DataStreams/IBlockInputStream.cpp index 23ba9ff2970..a6484c41b4f 100644 --- a/src/DataStreams/IBlockInputStream.cpp +++ b/src/DataStreams/IBlockInputStream.cpp @@ -4,7 +4,8 @@ #include #include #include -#include +#include +#include namespace ProfileEvents { @@ -359,8 +360,7 @@ Block IBlockInputStream::getExtremes() String IBlockInputStream::getTreeID() const { - std::stringstream s; - s.exceptions(std::ios::failbit); + WriteBufferFromOwnString s; s << getName(); if (!children.empty()) @@ -399,13 +399,13 @@ size_t IBlockInputStream::checkDepthImpl(size_t max_depth, size_t level) const } -void IBlockInputStream::dumpTree(std::ostream & ostr, size_t indent, size_t multiplier) const +void IBlockInputStream::dumpTree(WriteBuffer & ostr, size_t indent, size_t multiplier) const { ostr << String(indent, ' ') << getName(); if (multiplier > 1) ostr << " × " << multiplier; //ostr << ": " << getHeader().dumpStructure(); - ostr << std::endl; + ostr << '\n'; ++indent; /// If the subtree is repeated several times, then we output it once with the multiplier. diff --git a/src/DataStreams/IBlockInputStream.h b/src/DataStreams/IBlockInputStream.h index 3fbc3ce4bcd..4e314ef2980 100644 --- a/src/DataStreams/IBlockInputStream.h +++ b/src/DataStreams/IBlockInputStream.h @@ -95,7 +95,7 @@ public: virtual void readSuffix(); /// Must be called before `read()` and `readPrefix()`. - void dumpTree(std::ostream & ostr, size_t indent = 0, size_t multiplier = 1) const; + void dumpTree(WriteBuffer & ostr, size_t indent = 0, size_t multiplier = 1) const; /** Check the depth of the pipeline. * If max_depth is specified and the `depth` is greater - throw an exception. diff --git a/src/DataStreams/MergingSortedBlockInputStream.cpp b/src/DataStreams/MergingSortedBlockInputStream.cpp index 9c9213ef3cc..ee6b93dc8e4 100644 --- a/src/DataStreams/MergingSortedBlockInputStream.cpp +++ b/src/DataStreams/MergingSortedBlockInputStream.cpp @@ -1,6 +1,4 @@ #include -#include -#include #include diff --git a/src/DataStreams/MongoDBBlockInputStream.cpp b/src/DataStreams/MongoDBBlockInputStream.cpp index 25abdd909c4..5463d95151b 100644 --- a/src/DataStreams/MongoDBBlockInputStream.cpp +++ b/src/DataStreams/MongoDBBlockInputStream.cpp @@ -1,4 +1,3 @@ -#include #include #include diff --git a/src/DataTypes/DataTypeCustomSimpleAggregateFunction.cpp b/src/DataTypes/DataTypeCustomSimpleAggregateFunction.cpp index 3fe17a4bbfc..1d063dac697 100644 --- a/src/DataTypes/DataTypeCustomSimpleAggregateFunction.cpp +++ b/src/DataTypes/DataTypeCustomSimpleAggregateFunction.cpp @@ -32,8 +32,7 @@ static const std::vector supported_functions{"any", "anyLast", "min", String DataTypeCustomSimpleAggregateFunction::getName() const { - std::stringstream stream; - stream.exceptions(std::ios::failbit); + WriteBufferFromOwnString stream; stream << "SimpleAggregateFunction(" << function->getName(); if (!parameters.empty()) diff --git a/src/DataTypes/DataTypesDecimal.cpp b/src/DataTypes/DataTypesDecimal.cpp index bd4329f6f58..6c325c5d371 100644 --- a/src/DataTypes/DataTypesDecimal.cpp +++ b/src/DataTypes/DataTypesDecimal.cpp @@ -29,10 +29,7 @@ namespace ErrorCodes template std::string DataTypeDecimal::doGetName() const { - std::stringstream ss; - ss.exceptions(std::ios::failbit); - ss << "Decimal(" << this->precision << ", " << this->scale << ")"; - return ss.str(); + return fmt::format("Decimal({}, {})", this->precision, this->scale); } diff --git a/src/DataTypes/tests/gtest_data_type_get_common_type.cpp b/src/DataTypes/tests/gtest_data_type_get_common_type.cpp index 8212555e8bc..5c258f0b42e 100644 --- a/src/DataTypes/tests/gtest_data_type_get_common_type.cpp +++ b/src/DataTypes/tests/gtest_data_type_get_common_type.cpp @@ -26,7 +26,7 @@ static auto typeFromString(const std::string & str) static auto typesFromString(const std::string & str) { - std::istringstream data_types_stream(str); + std::istringstream data_types_stream(str); // STYLE_CHECK_ALLOW_STD_STRING_STREAM DataTypes data_types; std::string data_type; while (data_types_stream >> data_type) diff --git a/src/Databases/DatabaseOnDisk.cpp b/src/Databases/DatabaseOnDisk.cpp index 83e70a25f87..1e6b4019c4b 100644 --- a/src/Databases/DatabaseOnDisk.cpp +++ b/src/Databases/DatabaseOnDisk.cpp @@ -94,10 +94,9 @@ String getObjectDefinitionFromCreateQuery(const ASTPtr & query) if (!create) { - std::ostringstream query_stream; - query_stream.exceptions(std::ios::failbit); - formatAST(*query, query_stream, true); - throw Exception("Query '" + query_stream.str() + "' is not CREATE query", ErrorCodes::LOGICAL_ERROR); + WriteBufferFromOwnString query_buf; + formatAST(*query, query_buf, true); + throw Exception(ErrorCodes::LOGICAL_ERROR, "Query '{}' is not CREATE query", query_buf.str()); } if (!create->is_dictionary) @@ -121,11 +120,10 @@ String getObjectDefinitionFromCreateQuery(const ASTPtr & query) if (create->uuid != UUIDHelpers::Nil) create->table = TABLE_WITH_UUID_NAME_PLACEHOLDER; - std::ostringstream statement_stream; - statement_stream.exceptions(std::ios::failbit); - formatAST(*create, statement_stream, false); - statement_stream << '\n'; - return statement_stream.str(); + WriteBufferFromOwnString statement_buf; + formatAST(*create, statement_buf, false); + writeChar('\n', statement_buf); + return statement_buf.str(); } DatabaseOnDisk::DatabaseOnDisk( diff --git a/src/Databases/MySQL/MaterializeMySQLSyncThread.cpp b/src/Databases/MySQL/MaterializeMySQLSyncThread.cpp index 2a3de25c24f..7e42b2548b0 100644 --- a/src/Databases/MySQL/MaterializeMySQLSyncThread.cpp +++ b/src/Databases/MySQL/MaterializeMySQLSyncThread.cpp @@ -127,8 +127,7 @@ static String checkVariableAndGetVersion(const mysqlxx::Pool::Entry & connection } bool first = true; - std::stringstream error_message; - error_message.exceptions(std::ios::failbit); + WriteBufferFromOwnString error_message; error_message << "Illegal MySQL variables, the MaterializeMySQL engine requires "; for (const auto & [variable_name, variable_error_message] : variables_error_message) { @@ -239,8 +238,7 @@ static inline BlockOutputStreamPtr getTableOutput(const String & database_name, { const StoragePtr & storage = DatabaseCatalog::instance().getTable(StorageID(database_name, table_name), query_context); - std::stringstream insert_columns_str; - insert_columns_str.exceptions(std::ios::failbit); + WriteBufferFromOwnString insert_columns_str; const StorageInMemoryMetadata & storage_metadata = storage->getInMemoryMetadata(); const ColumnsDescription & storage_columns = storage_metadata.getColumns(); const NamesAndTypesList & insert_columns_names = insert_materialized ? storage_columns.getAllPhysical() : storage_columns.getOrdinary(); @@ -331,10 +329,9 @@ std::optional MaterializeMySQLSyncThread::prepareSynchroniz const auto & position_message = [&]() { - std::stringstream ss; - ss.exceptions(std::ios::failbit); - position.dump(ss); - return ss.str(); + WriteBufferFromOwnString buf; + position.dump(buf); + return buf.str(); }; LOG_INFO(log, "MySQL dump database position: \n {}", position_message()); } @@ -374,10 +371,9 @@ void MaterializeMySQLSyncThread::flushBuffersData(Buffers & buffers, Materialize const auto & position_message = [&]() { - std::stringstream ss; - ss.exceptions(std::ios::failbit); - client.getPosition().dump(ss); - return ss.str(); + WriteBufferFromOwnString buf; + client.getPosition().dump(buf); + return buf.str(); }; LOG_INFO(log, "MySQL executed position: \n {}", position_message()); } @@ -632,10 +628,9 @@ void MaterializeMySQLSyncThread::onEvent(Buffers & buffers, const BinlogEventPtr { const auto & dump_event_message = [&]() { - std::stringstream ss; - ss.exceptions(std::ios::failbit); - receive_event->dump(ss); - return ss.str(); + WriteBufferFromOwnString buf; + receive_event->dump(buf); + return buf.str(); }; LOG_DEBUG(log, "Skip MySQL event: \n {}", dump_event_message()); diff --git a/src/Dictionaries/DictionaryStructure.cpp b/src/Dictionaries/DictionaryStructure.cpp index 4c7cc5b4118..44924b69363 100644 --- a/src/Dictionaries/DictionaryStructure.cpp +++ b/src/Dictionaries/DictionaryStructure.cpp @@ -4,6 +4,7 @@ #include #include #include +#include #include #include @@ -230,8 +231,7 @@ std::string DictionaryStructure::getKeyDescription() const if (id) return "UInt64"; - std::ostringstream out; - out.exceptions(std::ios::failbit); + WriteBufferFromOwnString out; out << '('; diff --git a/src/Dictionaries/tests/gtest_dictionary_configuration.cpp b/src/Dictionaries/tests/gtest_dictionary_configuration.cpp index 62422124bd8..be0b0d7c6d2 100644 --- a/src/Dictionaries/tests/gtest_dictionary_configuration.cpp +++ b/src/Dictionaries/tests/gtest_dictionary_configuration.cpp @@ -18,7 +18,7 @@ static bool registered = false; static std::string configurationToString(const DictionaryConfigurationPtr & config) { const Poco::Util::XMLConfiguration * xml_config = dynamic_cast(config.get()); - std::ostringstream oss; + std::ostringstream oss; // STYLE_CHECK_ALLOW_STD_STRING_STREAM oss.exceptions(std::ios::failbit); xml_config->save(oss); return oss.str(); diff --git a/src/Functions/abtesting.cpp b/src/Functions/abtesting.cpp index 051b1f6f0ef..9f919051ee9 100644 --- a/src/Functions/abtesting.cpp +++ b/src/Functions/abtesting.cpp @@ -3,7 +3,6 @@ #if !defined(ARCADIA_BUILD) && USE_STATS #include -#include #include #include diff --git a/src/IO/CompressionMethod.cpp b/src/IO/CompressionMethod.cpp index a0a5e19f4fa..ec278b5d71f 100644 --- a/src/IO/CompressionMethod.cpp +++ b/src/IO/CompressionMethod.cpp @@ -1,11 +1,13 @@ #include -#include -#include -#include -#include #include #include +#include +#include +#include +#include +#include +#include #if !defined(ARCADIA_BUILD) # include @@ -14,7 +16,6 @@ namespace DB { - namespace ErrorCodes { extern const int NOT_IMPLEMENTED; @@ -25,10 +26,16 @@ std::string toContentEncodingName(CompressionMethod method) { switch (method) { - case CompressionMethod::Gzip: return "gzip"; - case CompressionMethod::Zlib: return "deflate"; - case CompressionMethod::Brotli: return "br"; - case CompressionMethod::None: return ""; + case CompressionMethod::Gzip: + return "gzip"; + case CompressionMethod::Zlib: + return "deflate"; + case CompressionMethod::Brotli: + return "br"; + case CompressionMethod::Xz: + return "xz"; + case CompressionMethod::None: + return ""; } __builtin_unreachable(); } @@ -52,20 +59,19 @@ CompressionMethod chooseCompressionMethod(const std::string & path, const std::s return CompressionMethod::Zlib; if (*method_str == "brotli" || *method_str == "br") return CompressionMethod::Brotli; + if (*method_str == "LZMA" || *method_str == "xz") + return CompressionMethod::Xz; if (hint.empty() || hint == "auto" || hint == "none") return CompressionMethod::None; - throw Exception("Unknown compression method " + hint + ". Only 'auto', 'none', 'gzip', 'br' are supported as compression methods", + throw Exception( + "Unknown compression method " + hint + ". Only 'auto', 'none', 'gzip', 'br', 'xz' are supported as compression methods", ErrorCodes::NOT_IMPLEMENTED); } std::unique_ptr wrapReadBufferWithCompressionMethod( - std::unique_ptr nested, - CompressionMethod method, - size_t buf_size, - char * existing_memory, - size_t alignment) + std::unique_ptr nested, CompressionMethod method, size_t buf_size, char * existing_memory, size_t alignment) { if (method == CompressionMethod::Gzip || method == CompressionMethod::Zlib) return std::make_unique(std::move(nested), method, buf_size, existing_memory, alignment); @@ -73,6 +79,8 @@ std::unique_ptr wrapReadBufferWithCompressionMethod( if (method == CompressionMethod::Brotli) return std::make_unique(std::move(nested), buf_size, existing_memory, alignment); #endif + if (method == CompressionMethod::Xz) + return std::make_unique(std::move(nested), buf_size, existing_memory, alignment); if (method == CompressionMethod::None) return nested; @@ -82,12 +90,7 @@ std::unique_ptr wrapReadBufferWithCompressionMethod( std::unique_ptr wrapWriteBufferWithCompressionMethod( - std::unique_ptr nested, - CompressionMethod method, - int level, - size_t buf_size, - char * existing_memory, - size_t alignment) + std::unique_ptr nested, CompressionMethod method, int level, size_t buf_size, char * existing_memory, size_t alignment) { if (method == DB::CompressionMethod::Gzip || method == CompressionMethod::Zlib) return std::make_unique(std::move(nested), method, level, buf_size, existing_memory, alignment); @@ -96,6 +99,8 @@ std::unique_ptr wrapWriteBufferWithCompressionMethod( if (method == DB::CompressionMethod::Brotli) return std::make_unique(std::move(nested), level, buf_size, existing_memory, alignment); #endif + if (method == CompressionMethod::Xz) + return std::make_unique(std::move(nested), level, buf_size, existing_memory, alignment); if (method == CompressionMethod::None) return nested; diff --git a/src/IO/CompressionMethod.h b/src/IO/CompressionMethod.h index 64c2ba3341f..5b0d4330011 100644 --- a/src/IO/CompressionMethod.h +++ b/src/IO/CompressionMethod.h @@ -1,14 +1,13 @@ #pragma once -#include #include +#include #include namespace DB { - class ReadBuffer; class WriteBuffer; @@ -26,6 +25,9 @@ enum class CompressionMethod /// DEFLATE compression with zlib header and Adler32 checksum. /// This option corresponds to HTTP Content-Encoding: deflate. Zlib, + /// LZMA2-based content compression + /// This option corresponds to HTTP Content-Encoding: xz + Xz, Brotli }; diff --git a/src/IO/HTTPCommon.cpp b/src/IO/HTTPCommon.cpp index 04fec145775..fa30afbd54f 100644 --- a/src/IO/HTTPCommon.cpp +++ b/src/IO/HTTPCommon.cpp @@ -239,7 +239,7 @@ void assertResponseIsOk(const Poco::Net::HTTPRequest & request, Poco::Net::HTTPR if (!(status == Poco::Net::HTTPResponse::HTTP_OK || (isRedirect(status) && allow_redirects))) { - std::stringstream error_message; + std::stringstream error_message; // STYLE_CHECK_ALLOW_STD_STRING_STREAM error_message.exceptions(std::ios::failbit); error_message << "Received error from remote server " << request.getURI() << ". HTTP status code: " << status << " " << response.getReason() << ", body: " << istr.rdbuf(); diff --git a/src/IO/LZMADeflatingWriteBuffer.cpp b/src/IO/LZMADeflatingWriteBuffer.cpp new file mode 100644 index 00000000000..f309aaa870e --- /dev/null +++ b/src/IO/LZMADeflatingWriteBuffer.cpp @@ -0,0 +1,125 @@ +#include + + +namespace DB +{ +namespace ErrorCodes +{ + extern const int LZMA_STREAM_ENCODER_FAILED; +} + +LZMADeflatingWriteBuffer::LZMADeflatingWriteBuffer( + std::unique_ptr out_, int compression_level, size_t buf_size, char * existing_memory, size_t alignment) + : BufferWithOwnMemory(buf_size, existing_memory, alignment), out(std::move(out_)) +{ + + lstr = LZMA_STREAM_INIT; + lstr.allocator = nullptr; + lstr.next_in = nullptr; + lstr.avail_in = 0; + lstr.next_out = nullptr; + lstr.avail_out = 0; + + // options for further compression + lzma_options_lzma opt_lzma2; + if (lzma_lzma_preset(&opt_lzma2, compression_level)) + throw Exception(ErrorCodes::LZMA_STREAM_ENCODER_FAILED, "lzma preset failed: lzma version: {}", LZMA_VERSION_STRING); + + + // LZMA_FILTER_X86 - + // LZMA2 - codec for *.xz files compression; LZMA is not suitable for this purpose + // VLI - variable length integer (in *.xz most integers encoded as VLI) + // LZMA_VLI_UNKNOWN (UINT64_MAX) - VLI value to denote that the value is unknown + lzma_filter filters[] = { + {.id = LZMA_FILTER_X86, .options = nullptr}, + {.id = LZMA_FILTER_LZMA2, .options = &opt_lzma2}, + {.id = LZMA_VLI_UNKNOWN, .options = nullptr}, + }; + lzma_ret ret = lzma_stream_encoder(&lstr, filters, LZMA_CHECK_CRC64); + + if (ret != LZMA_OK) + throw Exception( + ErrorCodes::LZMA_STREAM_ENCODER_FAILED, + "lzma stream encoder init failed: error code: {} lzma version: {}", + ret, + LZMA_VERSION_STRING); +} + +LZMADeflatingWriteBuffer::~LZMADeflatingWriteBuffer() +{ + try + { + finish(); + + lzma_end(&lstr); + } + catch (...) + { + tryLogCurrentException(__PRETTY_FUNCTION__); + } +} + +void LZMADeflatingWriteBuffer::nextImpl() +{ + if (!offset()) + return; + + lstr.next_in = reinterpret_cast(working_buffer.begin()); + lstr.avail_in = offset(); + + lzma_action action = LZMA_RUN; + do + { + out->nextIfAtEnd(); + lstr.next_out = reinterpret_cast(out->position()); + lstr.avail_out = out->buffer().end() - out->position(); + + lzma_ret ret = lzma_code(&lstr, action); + out->position() = out->buffer().end() - lstr.avail_out; + + if (ret == LZMA_STREAM_END) + return; + + if (ret != LZMA_OK) + throw Exception( + ErrorCodes::LZMA_STREAM_ENCODER_FAILED, + "lzma stream encoding failed: error code: {}; lzma_version: {}", + ret, + LZMA_VERSION_STRING); + + } while (lstr.avail_in > 0 || lstr.avail_out == 0); +} + + +void LZMADeflatingWriteBuffer::finish() +{ + if (finished) + return; + + next(); + + do + { + out->nextIfAtEnd(); + lstr.next_out = reinterpret_cast(out->position()); + lstr.avail_out = out->buffer().end() - out->position(); + + lzma_ret ret = lzma_code(&lstr, LZMA_FINISH); + out->position() = out->buffer().end() - lstr.avail_out; + + if (ret == LZMA_STREAM_END) + { + finished = true; + return; + } + + if (ret != LZMA_OK) + throw Exception( + ErrorCodes::LZMA_STREAM_ENCODER_FAILED, + "lzma stream encoding failed: error code: {}; lzma version: {}", + ret, + LZMA_VERSION_STRING); + + } while (lstr.avail_out == 0); +} +} diff --git a/src/IO/LZMADeflatingWriteBuffer.h b/src/IO/LZMADeflatingWriteBuffer.h new file mode 100644 index 00000000000..085913f024d --- /dev/null +++ b/src/IO/LZMADeflatingWriteBuffer.h @@ -0,0 +1,32 @@ +#pragma once + +#include +#include + +#include + +namespace DB +{ +/// Performs compression using lzma library and writes compressed data to out_ WriteBuffer. +class LZMADeflatingWriteBuffer : public BufferWithOwnMemory +{ +public: + LZMADeflatingWriteBuffer( + std::unique_ptr out_, + int compression_level, + size_t buf_size = DBMS_DEFAULT_BUFFER_SIZE, + char * existing_memory = nullptr, + size_t alignment = 0); + + void finish(); + + ~LZMADeflatingWriteBuffer() override; + +private: + void nextImpl() override; + + std::unique_ptr out; + lzma_stream lstr; + bool finished = false; +}; +} diff --git a/src/IO/LZMAInflatingReadBuffer.cpp b/src/IO/LZMAInflatingReadBuffer.cpp new file mode 100644 index 00000000000..30359987390 --- /dev/null +++ b/src/IO/LZMAInflatingReadBuffer.cpp @@ -0,0 +1,89 @@ +#include + +namespace DB +{ +namespace ErrorCodes +{ + extern const int LZMA_STREAM_DECODER_FAILED; +} +LZMAInflatingReadBuffer::LZMAInflatingReadBuffer(std::unique_ptr in_, size_t buf_size, char * existing_memory, size_t alignment) + : BufferWithOwnMemory(buf_size, existing_memory, alignment), in(std::move(in_)), eof(false) +{ + lstr = LZMA_STREAM_INIT; + lstr.allocator = nullptr; + lstr.next_in = nullptr; + lstr.avail_in = 0; + lstr.next_out = nullptr; + lstr.avail_out = 0; + + // 500 mb + uint64_t memlimit = 500ULL << 20; + + lzma_ret ret = lzma_stream_decoder(&lstr, memlimit, LZMA_CONCATENATED); + // lzma does not provide api for converting error code to string unlike zlib + if (ret != LZMA_OK) + throw Exception( + ErrorCodes::LZMA_STREAM_DECODER_FAILED, + "lzma_stream_decoder initialization failed: error code: {}; lzma version: {}", + ret, + LZMA_VERSION_STRING); +} + +LZMAInflatingReadBuffer::~LZMAInflatingReadBuffer() +{ + lzma_end(&lstr); +} + +bool LZMAInflatingReadBuffer::nextImpl() +{ + if (eof) + return false; + + lzma_action action = LZMA_RUN; + + if (!lstr.avail_in) + { + in->nextIfAtEnd(); + lstr.next_in = reinterpret_cast(in->position()); + lstr.avail_in = in->buffer().end() - in->position(); + } + + if (in->eof()) + { + action = LZMA_FINISH; + } + + lstr.next_out = reinterpret_cast(internal_buffer.begin()); + lstr.avail_out = internal_buffer.size(); + + lzma_ret ret = lzma_code(&lstr, action); + in->position() = in->buffer().end() - lstr.avail_in; + working_buffer.resize(internal_buffer.size() - lstr.avail_out); + + if (ret == LZMA_STREAM_END) + { + if (in->eof()) + { + eof = true; + return working_buffer.size() != 0; + } + else + { + throw Exception( + ErrorCodes::LZMA_STREAM_DECODER_FAILED, + "lzma decoder finished, but input stream has not exceeded: error code: {}; lzma version: {}", + ret, + LZMA_VERSION_STRING); + } + } + + if (ret != LZMA_OK) + throw Exception( + ErrorCodes::LZMA_STREAM_DECODER_FAILED, + "lzma_stream_decoder failed: error code: error codeL {}; lzma version: {}", + ret, + LZMA_VERSION_STRING); + + return true; +} +} diff --git a/src/IO/LZMAInflatingReadBuffer.h b/src/IO/LZMAInflatingReadBuffer.h new file mode 100644 index 00000000000..6994b2f50ee --- /dev/null +++ b/src/IO/LZMAInflatingReadBuffer.h @@ -0,0 +1,33 @@ +#pragma once + +#include +#include + +#include + +namespace DB +{ +namespace ErrorCodes +{ +} + +class LZMAInflatingReadBuffer : public BufferWithOwnMemory +{ +public: + LZMAInflatingReadBuffer( + std::unique_ptr in_, + size_t buf_size = DBMS_DEFAULT_BUFFER_SIZE, + char * existing_memory = nullptr, + size_t alignment = 0); + + ~LZMAInflatingReadBuffer() override; + +private: + bool nextImpl() override; + + std::unique_ptr in; + lzma_stream lstr; + + bool eof; +}; +} diff --git a/src/IO/MySQLPacketPayloadReadBuffer.cpp b/src/IO/MySQLPacketPayloadReadBuffer.cpp index f6f899e0ac7..9ca7845b2ae 100644 --- a/src/IO/MySQLPacketPayloadReadBuffer.cpp +++ b/src/IO/MySQLPacketPayloadReadBuffer.cpp @@ -1,5 +1,4 @@ #include -#include namespace DB { diff --git a/src/IO/Operators.h b/src/IO/Operators.h index 7e276ba9248..d1500aedd22 100644 --- a/src/IO/Operators.h +++ b/src/IO/Operators.h @@ -45,7 +45,9 @@ struct BinaryManipReadBuffer : std::reference_wrapper { usin template WriteBuffer & operator<< (WriteBuffer & buf, const T & x) { writeText(x, buf); return buf; } /// If you do not use the manipulators, the string is displayed without an escape, as is. template <> inline WriteBuffer & operator<< (WriteBuffer & buf, const String & x) { writeString(x, buf); return buf; } +template <> inline WriteBuffer & operator<< (WriteBuffer & buf, const std::string_view & x) { writeString(StringRef(x), buf); return buf; } template <> inline WriteBuffer & operator<< (WriteBuffer & buf, const char & x) { writeChar(x, buf); return buf; } +template <> inline WriteBuffer & operator<< (WriteBuffer & buf, const pcg32_fast & x) { PcgSerializer::serializePcg32(x, buf); return buf; } inline WriteBuffer & operator<< (WriteBuffer & buf, const char * x) { writeCString(x, buf); return buf; } @@ -73,6 +75,7 @@ inline WriteBuffer & operator<< (WriteBuffer & buf, FlushManip) { buf.next(); re template ReadBuffer & operator>> (ReadBuffer & buf, T & x) { readText(x, buf); return buf; } template <> inline ReadBuffer & operator>> (ReadBuffer & buf, String & x) { readString(x, buf); return buf; } template <> inline ReadBuffer & operator>> (ReadBuffer & buf, char & x) { readChar(x, buf); return buf; } +template <> inline ReadBuffer & operator>> (ReadBuffer & buf, pcg32_fast & x) { PcgDeserializer::deserializePcg32(x, buf); return buf; } /// If you specify a string literal for reading, this will mean - make sure there is a sequence of bytes and skip it. inline ReadBuffer & operator>> (ReadBuffer & buf, const char * x) { assertString(x, buf); return buf; } diff --git a/src/IO/ReadHelpers.h b/src/IO/ReadHelpers.h index 0e91bc1dd8e..a4531ebc24d 100644 --- a/src/IO/ReadHelpers.h +++ b/src/IO/ReadHelpers.h @@ -53,6 +53,7 @@ namespace ErrorCodes extern const int CANNOT_READ_ARRAY_FROM_TEXT; extern const int CANNOT_PARSE_NUMBER; extern const int ILLEGAL_TYPE_OF_ARGUMENT; + extern const int INCORRECT_DATA; } /// Helper functions for formatted input. @@ -1233,4 +1234,23 @@ void saveUpToPosition(ReadBuffer & in, Memory<> & memory, char * current); */ bool loadAtPosition(ReadBuffer & in, Memory<> & memory, char * & current); + +struct PcgDeserializer +{ + static void deserializePcg32(const pcg32_fast & rng, ReadBuffer & buf) + { + decltype(rng.state_) multiplier, increment, state; + readText(multiplier, buf); + assertChar(' ', buf); + readText(increment, buf); + assertChar(' ', buf); + readText(state, buf); + + if (multiplier != rng.multiplier()) + throw Exception(ErrorCodes::INCORRECT_DATA, "Incorrect multiplier in pcg32: expected {}, got {}", rng.multiplier(), multiplier); + if (increment != rng.increment()) + throw Exception(ErrorCodes::INCORRECT_DATA, "Incorrect increment in pcg32: expected {}, got {}", rng.increment(), increment); + } +}; + } diff --git a/src/IO/S3/PocoHTTPClient.cpp b/src/IO/S3/PocoHTTPClient.cpp index 49ccb6dc1b3..4a5b79e31ea 100644 --- a/src/IO/S3/PocoHTTPClient.cpp +++ b/src/IO/S3/PocoHTTPClient.cpp @@ -7,6 +7,8 @@ #include #include #include +#include +#include #include #include #include @@ -247,8 +249,7 @@ void PocoHTTPClient::makeRequestInternal( response->SetResponseCode(static_cast(status_code)); response->SetContentType(poco_response.getContentType()); - std::stringstream headers_ss; - headers_ss.exceptions(std::ios::failbit); + WriteBufferFromOwnString headers_ss; for (const auto & [header_name, header_value] : poco_response) { response->AddHeader(header_name, header_value); diff --git a/src/IO/WriteBufferFromS3.cpp b/src/IO/WriteBufferFromS3.cpp index 1ca8d8988d9..a32aa4acdc9 100644 --- a/src/IO/WriteBufferFromS3.cpp +++ b/src/IO/WriteBufferFromS3.cpp @@ -9,6 +9,7 @@ # include # include # include +# include # include # include # include @@ -167,6 +168,25 @@ void WriteBufferFromS3::complete() { if (is_multipart) { + if (part_tags.empty()) + { + LOG_DEBUG(log, "Multipart upload has no data. Aborting it. Bucket: {}, Key: {}, Upload_id: {}", bucket, key, upload_id); + + Aws::S3::Model::AbortMultipartUploadRequest req; + req.SetBucket(bucket); + req.SetKey(key); + req.SetUploadId(upload_id); + + auto outcome = client_ptr->AbortMultipartUpload(req); + + if (outcome.IsSuccess()) + LOG_DEBUG(log, "Aborting multipart upload completed. Upload_id: {}", upload_id); + else + throw Exception(outcome.GetError().GetMessage(), ErrorCodes::S3_ERROR); + + return; + } + LOG_DEBUG(log, "Completing multipart upload. Bucket: {}, Key: {}, Upload_id: {}", bucket, key, upload_id); Aws::S3::Model::CompleteMultipartUploadRequest req; @@ -174,18 +194,15 @@ void WriteBufferFromS3::complete() req.SetKey(key); req.SetUploadId(upload_id); - if (!part_tags.empty()) + Aws::S3::Model::CompletedMultipartUpload multipart_upload; + for (size_t i = 0; i < part_tags.size(); ++i) { - Aws::S3::Model::CompletedMultipartUpload multipart_upload; - for (size_t i = 0; i < part_tags.size(); ++i) - { - Aws::S3::Model::CompletedPart part; - multipart_upload.AddParts(part.WithETag(part_tags[i]).WithPartNumber(i + 1)); - } - - req.SetMultipartUpload(multipart_upload); + Aws::S3::Model::CompletedPart part; + multipart_upload.AddParts(part.WithETag(part_tags[i]).WithPartNumber(i + 1)); } + req.SetMultipartUpload(multipart_upload); + auto outcome = client_ptr->CompleteMultipartUpload(req); if (outcome.IsSuccess()) diff --git a/src/IO/WriteBufferFromVector.h b/src/IO/WriteBufferFromVector.h index 047e16643ce..54f43b6b591 100644 --- a/src/IO/WriteBufferFromVector.h +++ b/src/IO/WriteBufferFromVector.h @@ -36,8 +36,10 @@ private: throw Exception("WriteBufferFromVector is finished", ErrorCodes::CANNOT_WRITE_AFTER_END_OF_BUFFER); size_t old_size = vector.size(); + /// pos may not be equal to vector.data() + old_size, because WriteBuffer::next() can be used to flush data + size_t pos_offset = pos - reinterpret_cast(vector.data()); vector.resize(old_size * size_multiplier); - internal_buffer = Buffer(reinterpret_cast(vector.data() + old_size), reinterpret_cast(vector.data() + vector.size())); + internal_buffer = Buffer(reinterpret_cast(vector.data() + pos_offset), reinterpret_cast(vector.data() + vector.size())); working_buffer = internal_buffer; } diff --git a/src/IO/WriteHelpers.cpp b/src/IO/WriteHelpers.cpp index a0a2a45c791..61bfc281050 100644 --- a/src/IO/WriteHelpers.cpp +++ b/src/IO/WriteHelpers.cpp @@ -89,4 +89,12 @@ void writeProbablyBackQuotedStringMySQL(const StringRef & s, WriteBuffer & buf) writeProbablyQuotedStringImpl(s, buf, [](const StringRef & s_, WriteBuffer & buf_) { return writeBackQuotedStringMySQL(s_, buf_); }); } +void writePointerHex(const void * ptr, WriteBuffer & buf) +{ + writeString("0x", buf); + char hex_str[2 * sizeof(ptr)]; + writeHexUIntLowercase(reinterpret_cast(ptr), hex_str); + buf.write(hex_str, 2 * sizeof(ptr)); +} + } diff --git a/src/IO/WriteHelpers.h b/src/IO/WriteHelpers.h index 9d517913582..dcf4e50e5af 100644 --- a/src/IO/WriteHelpers.h +++ b/src/IO/WriteHelpers.h @@ -1093,4 +1093,18 @@ writeBinaryBigEndian(T x, WriteBuffer & buf) /// Assuming little endian archi writePODBinary(x, buf); } +struct PcgSerializer +{ + static void serializePcg32(const pcg32_fast & rng, WriteBuffer & buf) + { + writeText(rng.multiplier(), buf); + writeChar(' ', buf); + writeText(rng.increment(), buf); + writeChar(' ', buf); + writeText(rng.state_, buf); + } +}; + +void writePointerHex(const void * ptr, WriteBuffer & buf); + } diff --git a/src/IO/tests/CMakeLists.txt b/src/IO/tests/CMakeLists.txt index dfbbfa77853..b54785bf370 100644 --- a/src/IO/tests/CMakeLists.txt +++ b/src/IO/tests/CMakeLists.txt @@ -65,6 +65,9 @@ endif () add_executable (zlib_buffers zlib_buffers.cpp) target_link_libraries (zlib_buffers PRIVATE clickhouse_common_io) +add_executable (lzma_buffers lzma_buffers.cpp) +target_link_libraries (lzma_buffers PRIVATE clickhouse_common_io) + add_executable (limit_read_buffer limit_read_buffer.cpp) target_link_libraries (limit_read_buffer PRIVATE clickhouse_common_io) diff --git a/src/IO/tests/gtest_bit_io.cpp b/src/IO/tests/gtest_bit_io.cpp index dce146eaad7..2302a6c6135 100644 --- a/src/IO/tests/gtest_bit_io.cpp +++ b/src/IO/tests/gtest_bit_io.cpp @@ -76,7 +76,7 @@ std::string dumpContents(const T& container, const size_t cols_in_row = 8) { - std::stringstream sstr; + std::stringstream sstr; // STYLE_CHECK_ALLOW_STD_STRING_STREAM sstr.exceptions(std::ios::failbit); dumpBuffer(std::begin(container), std::end(container), &sstr, col_sep, row_sep, cols_in_row); diff --git a/src/IO/tests/hashing_read_buffer.cpp b/src/IO/tests/hashing_read_buffer.cpp index a1140160c32..67b3d61b1dc 100644 --- a/src/IO/tests/hashing_read_buffer.cpp +++ b/src/IO/tests/hashing_read_buffer.cpp @@ -22,7 +22,7 @@ static void test(size_t data_size) for (size_t read_buffer_block_size : block_sizes) { std::cout << "block size " << read_buffer_block_size << std::endl; - std::stringstream io; + std::stringstream io; // STYLE_CHECK_ALLOW_STD_STRING_STREAM io.exceptions(std::ios::failbit); DB::WriteBufferFromOStream out_impl(io); DB::HashingWriteBuffer out(out_impl); diff --git a/src/IO/tests/limit_read_buffer2.cpp b/src/IO/tests/limit_read_buffer2.cpp index 416eae0966b..d70ad832020 100644 --- a/src/IO/tests/limit_read_buffer2.cpp +++ b/src/IO/tests/limit_read_buffer2.cpp @@ -20,7 +20,7 @@ try { using namespace DB; - std::stringstream s; + std::stringstream s; // STYLE_CHECK_ALLOW_STD_STRING_STREAM s.exceptions(std::ios::failbit); { diff --git a/src/IO/tests/lzma_buffers.cpp b/src/IO/tests/lzma_buffers.cpp new file mode 100644 index 00000000000..7eb6bf8b81c --- /dev/null +++ b/src/IO/tests/lzma_buffers.cpp @@ -0,0 +1,64 @@ +#include +#include + +#include +#include +#include +#include +#include +#include +#include + +int main(int, char **) +try +{ + std::cout << std::fixed << std::setprecision(2); + + size_t n = 10000000; + Stopwatch stopwatch; + + { + auto buf + = std::make_unique("test_lzma_buffers.xz", DBMS_DEFAULT_BUFFER_SIZE, O_WRONLY | O_CREAT | O_TRUNC); + DB::LZMADeflatingWriteBuffer lzma_buf(std::move(buf), /*compression level*/ 3); + + stopwatch.restart(); + for (size_t i = 0; i < n; ++i) + { + DB::writeIntText(i, lzma_buf); + DB::writeChar('\t', lzma_buf); + } + lzma_buf.finish(); + + stopwatch.stop(); + + std::cout << "Writing done. Elapsed: " << stopwatch.elapsedSeconds() << " s." + << ", " << (lzma_buf.count() / stopwatch.elapsedSeconds() / 1000000) << " MB/s" << std::endl; + } + + { + auto buf = std::make_unique("test_lzma_buffers.xz"); + DB::LZMAInflatingReadBuffer lzma_buf(std::move(buf)); + + stopwatch.restart(); + for (size_t i = 0; i < n; ++i) + { + size_t x; + DB::readIntText(x, lzma_buf); + lzma_buf.ignore(); + + if (x != i) + throw DB::Exception("Failed!, read: " + std::to_string(x) + ", expected: " + std::to_string(i), 0); + } + stopwatch.stop(); + std::cout << "Reading done. Elapsed: " << stopwatch.elapsedSeconds() << " s." + << ", " << (lzma_buf.count() / stopwatch.elapsedSeconds() / 1000000) << " MB/s" << std::endl; + } + + return 0; +} +catch (const DB::Exception & e) +{ + std::cerr << e.what() << ", " << e.displayText() << std::endl; + return 1; +} diff --git a/src/IO/tests/write_buffer.cpp b/src/IO/tests/write_buffer.cpp index c0e9150d372..09b46f810bf 100644 --- a/src/IO/tests/write_buffer.cpp +++ b/src/IO/tests/write_buffer.cpp @@ -16,7 +16,7 @@ int main(int, char **) DB::String c = "вася пе\tтя"; DB::String d = "'xyz\\"; - std::stringstream s; + std::stringstream s; // STYLE_CHECK_ALLOW_STD_STRING_STREAM s.exceptions(std::ios::failbit); { diff --git a/src/IO/ya.make b/src/IO/ya.make index f935a00f385..a4d406d73ce 100644 --- a/src/IO/ya.make +++ b/src/IO/ya.make @@ -20,6 +20,8 @@ SRCS( HTTPCommon.cpp HashingWriteBuffer.cpp HexWriteBuffer.cpp + LZMADeflatingWriteBuffer.cpp + LZMAInflatingReadBuffer.cpp LimitReadBuffer.cpp MMapReadBufferFromFile.cpp MMapReadBufferFromFileDescriptor.cpp diff --git a/src/Interpreters/AddDefaultDatabaseVisitor.h b/src/Interpreters/AddDefaultDatabaseVisitor.h index bb684c5547a..7b72374c9c6 100644 --- a/src/Interpreters/AddDefaultDatabaseVisitor.h +++ b/src/Interpreters/AddDefaultDatabaseVisitor.h @@ -25,7 +25,7 @@ class AddDefaultDatabaseVisitor { public: explicit AddDefaultDatabaseVisitor( - const String & database_name_, bool only_replace_current_database_function_ = false, std::ostream * ostr_ = nullptr) + const String & database_name_, bool only_replace_current_database_function_ = false, WriteBuffer * ostr_ = nullptr) : database_name(database_name_) , only_replace_current_database_function(only_replace_current_database_function_) , visit_depth(0) @@ -66,7 +66,7 @@ private: const String database_name; bool only_replace_current_database_function = false; mutable size_t visit_depth; - std::ostream * ostr; + WriteBuffer * ostr; void visit(ASTSelectWithUnionQuery & select, ASTPtr &) const { diff --git a/src/Interpreters/ClusterProxy/SelectStreamFactory.cpp b/src/Interpreters/ClusterProxy/SelectStreamFactory.cpp index bb966d57c4f..3017550aaf1 100644 --- a/src/Interpreters/ClusterProxy/SelectStreamFactory.cpp +++ b/src/Interpreters/ClusterProxy/SelectStreamFactory.cpp @@ -98,10 +98,9 @@ String formattedAST(const ASTPtr & ast) { if (!ast) return {}; - std::stringstream ss; - ss.exceptions(std::ios::failbit); - formatAST(*ast, ss, false, true); - return ss.str(); + WriteBufferFromOwnString buf; + formatAST(*ast, buf, false, true); + return buf.str(); } } diff --git a/src/Interpreters/Context.cpp b/src/Interpreters/Context.cpp index a22eacd9a49..57c76147b7b 100644 --- a/src/Interpreters/Context.cpp +++ b/src/Interpreters/Context.cpp @@ -302,9 +302,11 @@ struct ContextShared mutable std::mutex zookeeper_mutex; mutable zkutil::ZooKeeperPtr zookeeper; /// Client for ZooKeeper. + ConfigurationPtr zookeeper_config; /// Stores zookeeper configs mutable std::mutex auxiliary_zookeepers_mutex; mutable std::map auxiliary_zookeepers; /// Map for auxiliary ZooKeeper clients. + ConfigurationPtr auxiliary_zookeepers_config; /// Stores auxiliary zookeepers configs String interserver_io_host; /// The host name by which this server is available for other servers. UInt16 interserver_io_port = 0; /// and port. @@ -364,8 +366,7 @@ struct ContextShared /// Initialized on demand (on distributed storages initialization) since Settings should be initialized std::unique_ptr clusters; ConfigurationPtr clusters_config; /// Stores updated configs - ConfigurationPtr zookeeper_config; /// Stores zookeeper configs - mutable std::mutex clusters_mutex; /// Guards clusters and clusters_config + mutable std::mutex clusters_mutex; /// Guards clusters and clusters_config #if USE_EMBEDDED_COMPILER std::shared_ptr compiled_expression_cache; @@ -1498,10 +1499,16 @@ zkutil::ZooKeeperPtr Context::getAuxiliaryZooKeeper(const String & name) const auto zookeeper = shared->auxiliary_zookeepers.find(name); if (zookeeper == shared->auxiliary_zookeepers.end()) { - if (!getConfigRef().has("auxiliary_zookeepers." + name)) - throw Exception(ErrorCodes::BAD_ARGUMENTS, "Unknown auxiliary ZooKeeper name '{}'. If it's required it can be added to the section in config.xml", name); + const auto & config = shared->auxiliary_zookeepers_config ? *shared->auxiliary_zookeepers_config : getConfigRef(); + if (!config.has("auxiliary_zookeepers." + name)) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "Unknown auxiliary ZooKeeper name '{}'. If it's required it can be added to the section in " + "config.xml", + name); - zookeeper->second = std::make_shared(getConfigRef(), "auxiliary_zookeepers." + name); + zookeeper + = shared->auxiliary_zookeepers.emplace(name, std::make_shared(config, "auxiliary_zookeepers." + name)).first; } else if (zookeeper->second->expired()) zookeeper->second = zookeeper->second->startNewSession(); @@ -1515,17 +1522,38 @@ void Context::resetZooKeeper() const shared->zookeeper.reset(); } +static void reloadZooKeeperIfChangedImpl(const ConfigurationPtr & config, const std::string & config_name, zkutil::ZooKeeperPtr & zk) +{ + if (!zk || zk->configChanged(*config, config_name)) + zk = std::make_shared(*config, config_name); +} + void Context::reloadZooKeeperIfChanged(const ConfigurationPtr & config) const { std::lock_guard lock(shared->zookeeper_mutex); shared->zookeeper_config = config; + reloadZooKeeperIfChangedImpl(config, "zookeeper", shared->zookeeper); +} - if (!shared->zookeeper || shared->zookeeper->configChanged(*config, "zookeeper")) +void Context::reloadAuxiliaryZooKeepersConfigIfChanged(const ConfigurationPtr & config) +{ + std::lock_guard lock(shared->auxiliary_zookeepers_mutex); + + shared->auxiliary_zookeepers_config = config; + + for (auto it = shared->auxiliary_zookeepers.begin(); it != shared->auxiliary_zookeepers.end();) { - shared->zookeeper = std::make_shared(*config, "zookeeper"); + if (!config->has("auxiliary_zookeepers." + it->first)) + it = shared->auxiliary_zookeepers.erase(it); + else + { + reloadZooKeeperIfChangedImpl(config, "auxiliary_zookeepers." + it->first, it->second); + ++it; + } } } + bool Context::hasZooKeeper() const { return getConfigRef().has("zookeeper"); @@ -1965,21 +1993,19 @@ void Context::checkCanBeDropped(const String & database, const String & table, c String size_str = formatReadableSizeWithDecimalSuffix(size); String max_size_to_drop_str = formatReadableSizeWithDecimalSuffix(max_size_to_drop); - std::stringstream ostr; - ostr.exceptions(std::ios::failbit); - - ostr << "Table or Partition in " << backQuoteIfNeed(database) << "." << backQuoteIfNeed(table) << " was not dropped.\n" - << "Reason:\n" - << "1. Size (" << size_str << ") is greater than max_[table/partition]_size_to_drop (" << max_size_to_drop_str << ")\n" - << "2. File '" << force_file.path() << "' intended to force DROP " - << (force_file_exists ? "exists but not writeable (could not be removed)" : "doesn't exist") << "\n"; - - ostr << "How to fix this:\n" - << "1. Either increase (or set to zero) max_[table/partition]_size_to_drop in server config\n" - << "2. Either create forcing file " << force_file.path() << " and make sure that ClickHouse has write permission for it.\n" - << "Example:\nsudo touch '" << force_file.path() << "' && sudo chmod 666 '" << force_file.path() << "'"; - - throw Exception(ostr.str(), ErrorCodes::TABLE_SIZE_EXCEEDS_MAX_DROP_SIZE_LIMIT); + throw Exception(ErrorCodes::TABLE_SIZE_EXCEEDS_MAX_DROP_SIZE_LIMIT, + "Table or Partition in {}.{} was not dropped.\nReason:\n" + "1. Size ({}) is greater than max_[table/partition]_size_to_drop ({})\n" + "2. File '{}' intended to force DROP {}\n" + "How to fix this:\n" + "1. Either increase (or set to zero) max_[table/partition]_size_to_drop in server config\n", + "2. Either create forcing file {} and make sure that ClickHouse has write permission for it.\n" + "Example:\nsudo touch '{}' && sudo chmod 666 '{}'", + backQuoteIfNeed(database), backQuoteIfNeed(table), + size_str, max_size_to_drop_str, + force_file.path(), force_file_exists ? "exists but not writeable (could not be removed)" : "doesn't exist", + force_file.path(), + force_file.path(), force_file.path()); } diff --git a/src/Interpreters/Context.h b/src/Interpreters/Context.h index ec42880309f..a2b61b29514 100644 --- a/src/Interpreters/Context.h +++ b/src/Interpreters/Context.h @@ -487,6 +487,9 @@ public: std::shared_ptr getZooKeeper() const; /// Same as above but return a zookeeper connection from auxiliary_zookeepers configuration entry. std::shared_ptr getAuxiliaryZooKeeper(const String & name) const; + + /// Set auxiliary zookeepers configuration at server starting or configuration reloading. + void reloadAuxiliaryZooKeepersConfigIfChanged(const ConfigurationPtr & config); /// Has ready or expired ZooKeeper bool hasZooKeeper() const; /// Reset current zookeeper session. Do not create a new one. diff --git a/src/Interpreters/ExpressionActions.cpp b/src/Interpreters/ExpressionActions.cpp index 75872553ec3..582937c6ac1 100644 --- a/src/Interpreters/ExpressionActions.cpp +++ b/src/Interpreters/ExpressionActions.cpp @@ -13,6 +13,7 @@ #include #include #include +#include #include #include #include @@ -453,8 +454,7 @@ void ExpressionAction::execute(Block & block, bool dry_run) const std::string ExpressionAction::toString() const { - std::stringstream ss; - ss.exceptions(std::ios::failbit); + WriteBufferFromOwnString ss; switch (type) { case ADD_COLUMN: @@ -550,8 +550,7 @@ void ExpressionActions::checkLimits(Block & block) const if (non_const_columns > settings.max_temporary_non_const_columns) { - std::stringstream list_of_non_const_columns; - list_of_non_const_columns.exceptions(std::ios::failbit); + WriteBufferFromOwnString list_of_non_const_columns; for (size_t i = 0, size = block.columns(); i < size; ++i) if (block.safeGetByPosition(i).column && !isColumnConst(*block.safeGetByPosition(i).column)) list_of_non_const_columns << "\n" << block.safeGetByPosition(i).name; @@ -922,8 +921,7 @@ void ExpressionActions::finalize(const Names & output_columns) std::string ExpressionActions::dumpActions() const { - std::stringstream ss; - ss.exceptions(std::ios::failbit); + WriteBufferFromOwnString ss; ss << "input:\n"; for (const auto & input_column : input_columns) @@ -1344,8 +1342,7 @@ void ExpressionActionsChain::finalize() std::string ExpressionActionsChain::dumpChain() const { - std::stringstream ss; - ss.exceptions(std::ios::failbit); + WriteBufferFromOwnString ss; for (size_t i = 0; i < steps.size(); ++i) { diff --git a/src/Interpreters/InDepthNodeVisitor.h b/src/Interpreters/InDepthNodeVisitor.h index 7b537f0daa0..7a793566cdd 100644 --- a/src/Interpreters/InDepthNodeVisitor.h +++ b/src/Interpreters/InDepthNodeVisitor.h @@ -16,7 +16,7 @@ class InDepthNodeVisitor public: using Data = typename Matcher::Data; - InDepthNodeVisitor(Data & data_, std::ostream * ostr_ = nullptr) + InDepthNodeVisitor(Data & data_, WriteBuffer * ostr_ = nullptr) : data(data_), visit_depth(0), ostr(ostr_) @@ -46,7 +46,7 @@ public: private: Data & data; size_t visit_depth; - std::ostream * ostr; + WriteBuffer * ostr; void visitChildren(T & ast) { diff --git a/src/Interpreters/InterpreterCreateQuery.cpp b/src/Interpreters/InterpreterCreateQuery.cpp index 286d5269a64..d6e4dc666e7 100644 --- a/src/Interpreters/InterpreterCreateQuery.cpp +++ b/src/Interpreters/InterpreterCreateQuery.cpp @@ -135,10 +135,7 @@ BlockIO InterpreterCreateQuery::createDatabase(ASTCreateQuery & create) else if ((create.columns_list && create.columns_list->indices && !create.columns_list->indices->children.empty())) { /// Currently, there are no database engines, that support any arguments. - std::stringstream ostr; - ostr.exceptions(std::ios::failbit); - formatAST(*create.storage, ostr, false, false); - throw Exception("Unknown database engine: " + ostr.str(), ErrorCodes::UNKNOWN_DATABASE_ENGINE); + throw Exception(ErrorCodes::UNKNOWN_DATABASE_ENGINE, "Unknown database engine: {}", serializeAST(*create.storage)); } if (create.storage->engine->name == "Atomic") @@ -182,11 +179,10 @@ BlockIO InterpreterCreateQuery::createDatabase(ASTCreateQuery & create) create.attach = true; create.if_not_exists = false; - std::ostringstream statement_stream; - statement_stream.exceptions(std::ios::failbit); - formatAST(create, statement_stream, false); - statement_stream << '\n'; - String statement = statement_stream.str(); + WriteBufferFromOwnString statement_buf; + formatAST(create, statement_buf, false); + writeChar('\n', statement_buf); + String statement = statement_buf.str(); /// Exclusive flag guarantees, that database is not created right now in another thread. WriteBufferFromFile out(metadata_file_tmp_path, statement.size(), O_WRONLY | O_CREAT | O_EXCL); diff --git a/src/Interpreters/InterpreterExplainQuery.cpp b/src/Interpreters/InterpreterExplainQuery.cpp index ed791f0d592..b13350d7ba2 100644 --- a/src/Interpreters/InterpreterExplainQuery.cpp +++ b/src/Interpreters/InterpreterExplainQuery.cpp @@ -11,10 +11,8 @@ #include #include #include -#include #include -#include #include #include @@ -222,15 +220,14 @@ BlockInputStreamPtr InterpreterExplainQuery::executeImpl() Block sample_block = getSampleBlock(); MutableColumns res_columns = sample_block.cloneEmptyColumns(); - std::stringstream ss; - ss.exceptions(std::ios::failbit); + WriteBufferFromOwnString buf; if (ast.getKind() == ASTExplainQuery::ParsedAST) { if (ast.getSettings()) throw Exception("Settings are not supported for EXPLAIN AST query.", ErrorCodes::UNKNOWN_SETTING); - dumpAST(*ast.getExplainedQuery(), ss); + dumpAST(*ast.getExplainedQuery(), buf); } else if (ast.getKind() == ASTExplainQuery::AnalyzedSyntax) { @@ -240,7 +237,7 @@ BlockInputStreamPtr InterpreterExplainQuery::executeImpl() ExplainAnalyzedSyntaxVisitor::Data data{.context = context}; ExplainAnalyzedSyntaxVisitor(data).visit(query); - ast.getExplainedQuery()->format(IAST::FormatSettings(ss, false)); + ast.getExplainedQuery()->format(IAST::FormatSettings(buf, false)); } else if (ast.getKind() == ASTExplainQuery::QueryPlan) { @@ -256,8 +253,7 @@ BlockInputStreamPtr InterpreterExplainQuery::executeImpl() if (settings.optimize) plan.optimize(); - WriteBufferFromOStream buffer(ss); - plan.explainPlan(buffer, settings.query_plan_options); + plan.explainPlan(buf, settings.query_plan_options); } else if (ast.getKind() == ASTExplainQuery::QueryPipeline) { @@ -271,8 +267,6 @@ BlockInputStreamPtr InterpreterExplainQuery::executeImpl() interpreter.buildQueryPlan(plan); auto pipeline = plan.buildQueryPipeline(); - WriteBufferFromOStream buffer(ss); - if (settings.graph) { /// Pipe holds QueryPlan, should not go out-of-scope @@ -280,17 +274,17 @@ BlockInputStreamPtr InterpreterExplainQuery::executeImpl() const auto & processors = pipe.getProcessors(); if (settings.compact) - printPipelineCompact(processors, buffer, settings.query_pipeline_options.header); + printPipelineCompact(processors, buf, settings.query_pipeline_options.header); else - printPipeline(processors, buffer); + printPipeline(processors, buf); } else { - plan.explainPipeline(buffer, settings.query_pipeline_options); + plan.explainPipeline(buf, settings.query_pipeline_options); } } - fillColumn(*res_columns[0], ss.str()); + fillColumn(*res_columns[0], buf.str()); return std::make_shared(sample_block.cloneWithColumns(std::move(res_columns))); } diff --git a/src/Interpreters/InterpreterShowAccessQuery.cpp b/src/Interpreters/InterpreterShowAccessQuery.cpp index 5f28c49c0bc..7bec7c411f0 100644 --- a/src/Interpreters/InterpreterShowAccessQuery.cpp +++ b/src/Interpreters/InterpreterShowAccessQuery.cpp @@ -34,13 +34,12 @@ BlockInputStreamPtr InterpreterShowAccessQuery::executeImpl() const /// Build the result column. MutableColumnPtr column = ColumnString::create(); - std::stringstream ss; - ss.exceptions(std::ios::failbit); + WriteBufferFromOwnString buf; for (const auto & query : queries) { - ss.str(""); - formatAST(*query, ss, false, true); - column->insert(ss.str()); + buf.restart(); + formatAST(*query, buf, false, true); + column->insert(buf.str()); } String desc = "ACCESS"; diff --git a/src/Interpreters/InterpreterShowCreateAccessEntityQuery.cpp b/src/Interpreters/InterpreterShowCreateAccessEntityQuery.cpp index 749a5811e13..3135b0cfdf2 100644 --- a/src/Interpreters/InterpreterShowCreateAccessEntityQuery.cpp +++ b/src/Interpreters/InterpreterShowCreateAccessEntityQuery.cpp @@ -26,7 +26,6 @@ #include #include #include -#include namespace DB @@ -238,21 +237,19 @@ BlockInputStreamPtr InterpreterShowCreateAccessEntityQuery::executeImpl() /// Build the result column. MutableColumnPtr column = ColumnString::create(); - std::stringstream create_query_ss; - create_query_ss.exceptions(std::ios::failbit); + WriteBufferFromOwnString create_query_buf; for (const auto & create_query : create_queries) { - formatAST(*create_query, create_query_ss, false, true); - column->insert(create_query_ss.str()); - create_query_ss.str(""); + formatAST(*create_query, create_query_buf, false, true); + column->insert(create_query_buf.str()); + create_query_buf.restart(); } /// Prepare description of the result column. - std::stringstream desc_ss; - desc_ss.exceptions(std::ios::failbit); + WriteBufferFromOwnString desc_buf; const auto & show_query = query_ptr->as(); - formatAST(show_query, desc_ss, false, true); - String desc = desc_ss.str(); + formatAST(show_query, desc_buf, false, true); + String desc = desc_buf.str(); String prefix = "SHOW "; if (startsWith(desc, prefix)) desc = desc.substr(prefix.length()); /// `desc` always starts with "SHOW ", so we can trim this prefix. diff --git a/src/Interpreters/InterpreterShowCreateQuery.cpp b/src/Interpreters/InterpreterShowCreateQuery.cpp index 8861914a68a..907523ce94b 100644 --- a/src/Interpreters/InterpreterShowCreateQuery.cpp +++ b/src/Interpreters/InterpreterShowCreateQuery.cpp @@ -1,5 +1,3 @@ -#include - #include #include #include @@ -78,10 +76,9 @@ BlockInputStreamPtr InterpreterShowCreateQuery::executeImpl() create.uuid = UUIDHelpers::Nil; } - std::stringstream stream; - stream.exceptions(std::ios::failbit); - formatAST(*create_query, stream, false, false); - String res = stream.str(); + WriteBufferFromOwnString buf; + formatAST(*create_query, buf, false, false); + String res = buf.str(); MutableColumnPtr column = ColumnString::create(); column->insert(res); diff --git a/src/Interpreters/InterpreterShowGrantsQuery.cpp b/src/Interpreters/InterpreterShowGrantsQuery.cpp index 7de51b6a7ee..a2ddc5eec27 100644 --- a/src/Interpreters/InterpreterShowGrantsQuery.cpp +++ b/src/Interpreters/InterpreterShowGrantsQuery.cpp @@ -118,21 +118,19 @@ BlockInputStreamPtr InterpreterShowGrantsQuery::executeImpl() /// Build the result column. MutableColumnPtr column = ColumnString::create(); - std::stringstream grant_ss; - grant_ss.exceptions(std::ios::failbit); + WriteBufferFromOwnString grant_buf; for (const auto & grant_query : grant_queries) { - grant_ss.str(""); - formatAST(*grant_query, grant_ss, false, true); - column->insert(grant_ss.str()); + grant_buf.restart(); + formatAST(*grant_query, grant_buf, false, true); + column->insert(grant_buf.str()); } /// Prepare description of the result column. - std::stringstream desc_ss; - desc_ss.exceptions(std::ios::failbit); + WriteBufferFromOwnString desc_buf; const auto & show_query = query_ptr->as(); - formatAST(show_query, desc_ss, false, true); - String desc = desc_ss.str(); + formatAST(show_query, desc_buf, false, true); + String desc = desc_buf.str(); String prefix = "SHOW "; if (desc.starts_with(prefix)) desc = desc.substr(prefix.length()); /// `desc` always starts with "SHOW ", so we can trim this prefix. diff --git a/src/Interpreters/InterpreterShowTablesQuery.cpp b/src/Interpreters/InterpreterShowTablesQuery.cpp index 8e67cecdd5e..861fa8c3505 100644 --- a/src/Interpreters/InterpreterShowTablesQuery.cpp +++ b/src/Interpreters/InterpreterShowTablesQuery.cpp @@ -6,8 +6,7 @@ #include #include #include -#include -#include +#include namespace DB @@ -32,8 +31,7 @@ String InterpreterShowTablesQuery::getRewrittenQuery() /// SHOW DATABASES if (query.databases) { - std::stringstream rewritten_query; - rewritten_query.exceptions(std::ios::failbit); + WriteBufferFromOwnString rewritten_query; rewritten_query << "SELECT name FROM system.databases"; if (!query.like.empty()) @@ -42,7 +40,7 @@ String InterpreterShowTablesQuery::getRewrittenQuery() << " WHERE name " << (query.not_like ? "NOT " : "") << (query.case_insensitive_like ? "ILIKE " : "LIKE ") - << std::quoted(query.like, '\''); + << DB::quote << query.like; } if (query.limit_length) @@ -54,8 +52,7 @@ String InterpreterShowTablesQuery::getRewrittenQuery() /// SHOW CLUSTER/CLUSTERS if (query.clusters) { - std::stringstream rewritten_query; - rewritten_query.exceptions(std::ios::failbit); + WriteBufferFromOwnString rewritten_query; rewritten_query << "SELECT DISTINCT cluster FROM system.clusters"; if (!query.like.empty()) @@ -64,7 +61,7 @@ String InterpreterShowTablesQuery::getRewrittenQuery() << " WHERE cluster " << (query.not_like ? "NOT " : "") << (query.case_insensitive_like ? "ILIKE " : "LIKE ") - << std::quoted(query.like, '\''); + << DB::quote << query.like; } if (query.limit_length) @@ -74,11 +71,10 @@ String InterpreterShowTablesQuery::getRewrittenQuery() } else if (query.cluster) { - std::stringstream rewritten_query; - rewritten_query.exceptions(std::ios::failbit); + WriteBufferFromOwnString rewritten_query; rewritten_query << "SELECT * FROM system.clusters"; - rewritten_query << " WHERE cluster = " << std::quoted(query.cluster_str, '\''); + rewritten_query << " WHERE cluster = " << DB::quote << query.cluster_str; return rewritten_query.str(); } @@ -89,8 +85,7 @@ String InterpreterShowTablesQuery::getRewrittenQuery() String database = context.resolveDatabase(query.from); DatabaseCatalog::instance().assertDatabaseExists(database); - std::stringstream rewritten_query; - rewritten_query.exceptions(std::ios::failbit); + WriteBufferFromOwnString rewritten_query; rewritten_query << "SELECT name FROM system."; if (query.dictionaries) @@ -107,14 +102,14 @@ String InterpreterShowTablesQuery::getRewrittenQuery() rewritten_query << "is_temporary"; } else - rewritten_query << "database = " << std::quoted(database, '\''); + rewritten_query << "database = " << DB::quote << database; if (!query.like.empty()) rewritten_query << " AND name " << (query.not_like ? "NOT " : "") << (query.case_insensitive_like ? "ILIKE " : "LIKE ") - << std::quoted(query.like, '\''); + << DB::quote << query.like; else if (query.where_expression) rewritten_query << " AND (" << query.where_expression << ")"; diff --git a/src/Interpreters/QueryAliasesVisitor.cpp b/src/Interpreters/QueryAliasesVisitor.cpp index 9de1d04990d..d395bfc20e9 100644 --- a/src/Interpreters/QueryAliasesVisitor.cpp +++ b/src/Interpreters/QueryAliasesVisitor.cpp @@ -1,6 +1,3 @@ -#include -#include - #include #include #include @@ -20,13 +17,12 @@ namespace ErrorCodes static String wrongAliasMessage(const ASTPtr & ast, const ASTPtr & prev_ast, const String & alias) { - std::stringstream message; - message.exceptions(std::ios::failbit); - message << "Different expressions with the same alias " << backQuoteIfNeed(alias) << ":" << std::endl; + WriteBufferFromOwnString message; + message << "Different expressions with the same alias " << backQuoteIfNeed(alias) << ":\n"; formatAST(*ast, message, false, true); - message << std::endl << "and" << std::endl; + message << "\nand\n"; formatAST(*prev_ast, message, false, true); - message << std::endl; + message << '\n'; return message.str(); } diff --git a/src/Interpreters/Set.cpp b/src/Interpreters/Set.cpp index 907cbaee243..8eeb939329b 100644 --- a/src/Interpreters/Set.cpp +++ b/src/Interpreters/Set.cpp @@ -342,11 +342,9 @@ void Set::checkColumnsNumber(size_t num_key_columns) const { if (data_types.size() != num_key_columns) { - std::stringstream message; - message.exceptions(std::ios::failbit); - message << "Number of columns in section IN doesn't match. " - << num_key_columns << " at left, " << data_types.size() << " at right."; - throw Exception(message.str(), ErrorCodes::NUMBER_OF_COLUMNS_DOESNT_MATCH); + throw Exception(ErrorCodes::NUMBER_OF_COLUMNS_DOESNT_MATCH, + "Number of columns in section IN doesn't match. {} at left, {} at right.", + num_key_columns, data_types.size()); } } diff --git a/src/Interpreters/TreeRewriter.cpp b/src/Interpreters/TreeRewriter.cpp index 51609cbaa23..b2dbd027191 100644 --- a/src/Interpreters/TreeRewriter.cpp +++ b/src/Interpreters/TreeRewriter.cpp @@ -552,8 +552,7 @@ void TreeRewriterResult::collectUsedColumns(const ASTPtr & query, bool is_select if (!unknown_required_source_columns.empty()) { - std::stringstream ss; - ss.exceptions(std::ios::failbit); + WriteBufferFromOwnString ss; ss << "Missing columns:"; for (const auto & name : unknown_required_source_columns) ss << " '" << name << "'"; diff --git a/src/Interpreters/executeQuery.cpp b/src/Interpreters/executeQuery.cpp index cdb3d9b7d7b..9de788a94d4 100644 --- a/src/Interpreters/executeQuery.cpp +++ b/src/Interpreters/executeQuery.cpp @@ -778,11 +778,9 @@ static std::tuple executeQueryImpl( if (!internal && res.in) { - std::stringstream log_str; - log_str.exceptions(std::ios::failbit); - log_str << "Query pipeline:\n"; - res.in->dumpTree(log_str); - LOG_DEBUG(&Poco::Logger::get("executeQuery"), log_str.str()); + WriteBufferFromOwnString msg_buf; + res.in->dumpTree(msg_buf); + LOG_DEBUG(&Poco::Logger::get("executeQuery"), "Query pipeline:\n{}", msg_buf.str()); } } } diff --git a/src/Parsers/ASTAlterQuery.cpp b/src/Parsers/ASTAlterQuery.cpp index d07e57eefae..7079c108711 100644 --- a/src/Parsers/ASTAlterQuery.cpp +++ b/src/Parsers/ASTAlterQuery.cpp @@ -1,4 +1,5 @@ #include +#include #include #include @@ -247,7 +248,7 @@ void ASTAlterCommand::formatImpl( << "PARTITION " << (settings.hilite ? hilite_none : ""); partition->formatImpl(settings, state, frame); settings.ostr << (settings.hilite ? hilite_keyword : "") - << " FROM " << (settings.hilite ? hilite_none : "") << std::quoted(from, '\''); + << " FROM " << (settings.hilite ? hilite_none : "") << DB::quote << from; } else if (type == ASTAlterCommand::FREEZE_PARTITION) { @@ -257,7 +258,7 @@ void ASTAlterCommand::formatImpl( if (!with_name.empty()) { settings.ostr << " " << (settings.hilite ? hilite_keyword : "") << "WITH NAME" << (settings.hilite ? hilite_none : "") - << " " << std::quoted(with_name, '\''); + << " " << DB::quote << with_name; } } else if (type == ASTAlterCommand::FREEZE_ALL) @@ -267,7 +268,7 @@ void ASTAlterCommand::formatImpl( if (!with_name.empty()) { settings.ostr << " " << (settings.hilite ? hilite_keyword : "") << "WITH NAME" << (settings.hilite ? hilite_none : "") - << " " << std::quoted(with_name, '\''); + << " " << DB::quote << with_name; } } else if (type == ASTAlterCommand::DELETE) diff --git a/src/Parsers/ASTAsterisk.cpp b/src/Parsers/ASTAsterisk.cpp index 95a63586685..ed733e62ca3 100644 --- a/src/Parsers/ASTAsterisk.cpp +++ b/src/Parsers/ASTAsterisk.cpp @@ -1,5 +1,6 @@ #include #include +#include namespace DB { diff --git a/src/Parsers/ASTColumnDeclaration.cpp b/src/Parsers/ASTColumnDeclaration.cpp index 27ece3e18c2..4c14230e926 100644 --- a/src/Parsers/ASTColumnDeclaration.cpp +++ b/src/Parsers/ASTColumnDeclaration.cpp @@ -1,5 +1,6 @@ #include #include +#include namespace DB diff --git a/src/Parsers/ASTColumnsMatcher.cpp b/src/Parsers/ASTColumnsMatcher.cpp index aab8e841981..45799cb7ffe 100644 --- a/src/Parsers/ASTColumnsMatcher.cpp +++ b/src/Parsers/ASTColumnsMatcher.cpp @@ -3,6 +3,7 @@ #include #include #include +#include namespace DB diff --git a/src/Parsers/ASTColumnsTransformers.cpp b/src/Parsers/ASTColumnsTransformers.cpp index a8f39079902..7c0fd0eb734 100644 --- a/src/Parsers/ASTColumnsTransformers.cpp +++ b/src/Parsers/ASTColumnsTransformers.cpp @@ -5,6 +5,7 @@ #include #include #include +#include namespace DB diff --git a/src/Parsers/ASTConstraintDeclaration.cpp b/src/Parsers/ASTConstraintDeclaration.cpp index 371bfa40f54..7d74837478c 100644 --- a/src/Parsers/ASTConstraintDeclaration.cpp +++ b/src/Parsers/ASTConstraintDeclaration.cpp @@ -1,5 +1,6 @@ #include #include +#include namespace DB diff --git a/src/Parsers/ASTCreateQuery.cpp b/src/Parsers/ASTCreateQuery.cpp index 4efe5476395..a193433c988 100644 --- a/src/Parsers/ASTCreateQuery.cpp +++ b/src/Parsers/ASTCreateQuery.cpp @@ -5,6 +5,7 @@ #include #include #include +#include namespace DB diff --git a/src/Parsers/ASTCreateQuotaQuery.cpp b/src/Parsers/ASTCreateQuotaQuery.cpp index 88516fb6eac..7e570b889e3 100644 --- a/src/Parsers/ASTCreateQuotaQuery.cpp +++ b/src/Parsers/ASTCreateQuotaQuery.cpp @@ -3,6 +3,7 @@ #include #include #include +#include namespace DB diff --git a/src/Parsers/ASTCreateRoleQuery.cpp b/src/Parsers/ASTCreateRoleQuery.cpp index 5ccfd9c6bd5..73b523a5bfe 100644 --- a/src/Parsers/ASTCreateRoleQuery.cpp +++ b/src/Parsers/ASTCreateRoleQuery.cpp @@ -1,6 +1,7 @@ #include #include #include +#include namespace DB diff --git a/src/Parsers/ASTCreateRowPolicyQuery.cpp b/src/Parsers/ASTCreateRowPolicyQuery.cpp index 6224b534851..30b001feeca 100644 --- a/src/Parsers/ASTCreateRowPolicyQuery.cpp +++ b/src/Parsers/ASTCreateRowPolicyQuery.cpp @@ -5,7 +5,7 @@ #include #include #include -#include +#include namespace DB @@ -62,14 +62,13 @@ namespace void formatForClauses(const std::vector> & conditions, bool alter, const IAST::FormatSettings & settings) { std::vector> conditions_as_strings; - std::stringstream temp_sstream; - temp_sstream.exceptions(std::ios::failbit); - IAST::FormatSettings temp_settings(temp_sstream, settings); + WriteBufferFromOwnString temp_buf; + IAST::FormatSettings temp_settings(temp_buf, settings); for (const auto & [condition_type, condition] : conditions) { formatConditionalExpression(condition, temp_settings); - conditions_as_strings.emplace_back(condition_type, temp_sstream.str()); - temp_sstream.str(""); + conditions_as_strings.emplace_back(condition_type, temp_buf.str()); + temp_buf.restart(); } boost::container::flat_set commands; diff --git a/src/Parsers/ASTCreateSettingsProfileQuery.cpp b/src/Parsers/ASTCreateSettingsProfileQuery.cpp index 77c2f1b22d7..84f8309462e 100644 --- a/src/Parsers/ASTCreateSettingsProfileQuery.cpp +++ b/src/Parsers/ASTCreateSettingsProfileQuery.cpp @@ -2,6 +2,7 @@ #include #include #include +#include namespace DB diff --git a/src/Parsers/ASTCreateUserQuery.cpp b/src/Parsers/ASTCreateUserQuery.cpp index 0ccc2232734..4b2aa70785a 100644 --- a/src/Parsers/ASTCreateUserQuery.cpp +++ b/src/Parsers/ASTCreateUserQuery.cpp @@ -3,6 +3,7 @@ #include #include #include +#include namespace DB diff --git a/src/Parsers/ASTDictionary.cpp b/src/Parsers/ASTDictionary.cpp index e0785f3bd49..878f6000aa9 100644 --- a/src/Parsers/ASTDictionary.cpp +++ b/src/Parsers/ASTDictionary.cpp @@ -1,5 +1,6 @@ #include #include +#include namespace DB { diff --git a/src/Parsers/ASTDictionaryAttributeDeclaration.cpp b/src/Parsers/ASTDictionaryAttributeDeclaration.cpp index 05ba48ace7b..e9c50839a98 100644 --- a/src/Parsers/ASTDictionaryAttributeDeclaration.cpp +++ b/src/Parsers/ASTDictionaryAttributeDeclaration.cpp @@ -1,5 +1,6 @@ #include #include +#include namespace DB diff --git a/src/Parsers/ASTDropAccessEntityQuery.cpp b/src/Parsers/ASTDropAccessEntityQuery.cpp index fe98d8b4158..1df176c24ec 100644 --- a/src/Parsers/ASTDropAccessEntityQuery.cpp +++ b/src/Parsers/ASTDropAccessEntityQuery.cpp @@ -1,6 +1,7 @@ #include #include #include +#include namespace DB diff --git a/src/Parsers/ASTDropQuery.cpp b/src/Parsers/ASTDropQuery.cpp index 0ced4a8ea96..b09b588ca6e 100644 --- a/src/Parsers/ASTDropQuery.cpp +++ b/src/Parsers/ASTDropQuery.cpp @@ -1,5 +1,6 @@ #include #include +#include namespace DB diff --git a/src/Parsers/ASTExpressionList.cpp b/src/Parsers/ASTExpressionList.cpp index de38e1fd7ea..2724465537f 100644 --- a/src/Parsers/ASTExpressionList.cpp +++ b/src/Parsers/ASTExpressionList.cpp @@ -1,4 +1,5 @@ #include +#include namespace DB diff --git a/src/Parsers/ASTFunction.cpp b/src/Parsers/ASTFunction.cpp index 66565eeaf8f..76a52b8c641 100644 --- a/src/Parsers/ASTFunction.cpp +++ b/src/Parsers/ASTFunction.cpp @@ -6,6 +6,7 @@ #include #include #include +#include namespace DB diff --git a/src/Parsers/ASTFunctionWithKeyValueArguments.cpp b/src/Parsers/ASTFunctionWithKeyValueArguments.cpp index 5f1e78b61da..d94490ab8b3 100644 --- a/src/Parsers/ASTFunctionWithKeyValueArguments.cpp +++ b/src/Parsers/ASTFunctionWithKeyValueArguments.cpp @@ -2,6 +2,7 @@ #include #include +#include namespace DB { diff --git a/src/Parsers/ASTGrantQuery.cpp b/src/Parsers/ASTGrantQuery.cpp index fefebc4ae4a..2610836c759 100644 --- a/src/Parsers/ASTGrantQuery.cpp +++ b/src/Parsers/ASTGrantQuery.cpp @@ -1,6 +1,7 @@ #include #include #include +#include namespace DB diff --git a/src/Parsers/ASTIdentifier.cpp b/src/Parsers/ASTIdentifier.cpp index 5a66bc7891d..2717edd3b91 100644 --- a/src/Parsers/ASTIdentifier.cpp +++ b/src/Parsers/ASTIdentifier.cpp @@ -1,10 +1,10 @@ #include -#include #include #include #include #include +#include namespace DB diff --git a/src/Parsers/ASTIndexDeclaration.cpp b/src/Parsers/ASTIndexDeclaration.cpp index e89f9bf26ed..0e8f0d0f7e8 100644 --- a/src/Parsers/ASTIndexDeclaration.cpp +++ b/src/Parsers/ASTIndexDeclaration.cpp @@ -1,5 +1,6 @@ #include #include +#include namespace DB diff --git a/src/Parsers/ASTInsertQuery.cpp b/src/Parsers/ASTInsertQuery.cpp index dc9b9f092ac..4096c484059 100644 --- a/src/Parsers/ASTInsertQuery.cpp +++ b/src/Parsers/ASTInsertQuery.cpp @@ -3,6 +3,7 @@ #include #include #include +#include namespace DB diff --git a/src/Parsers/ASTKillQueryQuery.cpp b/src/Parsers/ASTKillQueryQuery.cpp index 293b95b93bf..72bdd7d6b0b 100644 --- a/src/Parsers/ASTKillQueryQuery.cpp +++ b/src/Parsers/ASTKillQueryQuery.cpp @@ -1,4 +1,5 @@ #include +#include namespace DB { diff --git a/src/Parsers/ASTLiteral.cpp b/src/Parsers/ASTLiteral.cpp index cd9c389f336..ed6790499fb 100644 --- a/src/Parsers/ASTLiteral.cpp +++ b/src/Parsers/ASTLiteral.cpp @@ -73,4 +73,9 @@ void ASTLiteral::appendColumnNameImpl(WriteBuffer & ostr) const } } +void ASTLiteral::formatImplWithoutAlias(const FormatSettings & settings, IAST::FormatState &, IAST::FormatStateStacked) const +{ + settings.ostr << applyVisitor(FieldVisitorToString(), value); +} + } diff --git a/src/Parsers/ASTLiteral.h b/src/Parsers/ASTLiteral.h index 18f440a81a4..672bc6ddc3e 100644 --- a/src/Parsers/ASTLiteral.h +++ b/src/Parsers/ASTLiteral.h @@ -43,10 +43,7 @@ public: void updateTreeHashImpl(SipHash & hash_state) const override; protected: - void formatImplWithoutAlias(const FormatSettings & settings, FormatState &, FormatStateStacked) const override - { - settings.ostr << applyVisitor(FieldVisitorToString(), value); - } + void formatImplWithoutAlias(const FormatSettings & settings, FormatState &, FormatStateStacked) const override; void appendColumnNameImpl(WriteBuffer & ostr) const override; }; diff --git a/src/Parsers/ASTNameTypePair.cpp b/src/Parsers/ASTNameTypePair.cpp index 80b0f7f8ec6..e4066081a9b 100644 --- a/src/Parsers/ASTNameTypePair.cpp +++ b/src/Parsers/ASTNameTypePair.cpp @@ -1,5 +1,6 @@ #include #include +#include namespace DB diff --git a/src/Parsers/ASTOptimizeQuery.cpp b/src/Parsers/ASTOptimizeQuery.cpp index 92968f2b277..ae83952899d 100644 --- a/src/Parsers/ASTOptimizeQuery.cpp +++ b/src/Parsers/ASTOptimizeQuery.cpp @@ -1,5 +1,6 @@ #include #include +#include namespace DB { diff --git a/src/Parsers/ASTOrderByElement.cpp b/src/Parsers/ASTOrderByElement.cpp index 48290b2669a..884d69a18e3 100644 --- a/src/Parsers/ASTOrderByElement.cpp +++ b/src/Parsers/ASTOrderByElement.cpp @@ -1,6 +1,7 @@ #include #include #include +#include namespace DB diff --git a/src/Parsers/ASTPartition.cpp b/src/Parsers/ASTPartition.cpp index d24575b7f43..06bfe4f5217 100644 --- a/src/Parsers/ASTPartition.cpp +++ b/src/Parsers/ASTPartition.cpp @@ -1,5 +1,6 @@ #include #include +#include namespace DB { diff --git a/src/Parsers/ASTQualifiedAsterisk.cpp b/src/Parsers/ASTQualifiedAsterisk.cpp index 0cda01cecac..2491dcb36b7 100644 --- a/src/Parsers/ASTQualifiedAsterisk.cpp +++ b/src/Parsers/ASTQualifiedAsterisk.cpp @@ -1,5 +1,6 @@ #include #include +#include namespace DB { diff --git a/src/Parsers/ASTQueryParameter.cpp b/src/Parsers/ASTQueryParameter.cpp index 915ecd5e7e4..c10cced23ce 100644 --- a/src/Parsers/ASTQueryParameter.cpp +++ b/src/Parsers/ASTQueryParameter.cpp @@ -1,6 +1,7 @@ #include #include #include +#include namespace DB diff --git a/src/Parsers/ASTQueryWithOnCluster.cpp b/src/Parsers/ASTQueryWithOnCluster.cpp index cb77d351d54..60e96b1dbe1 100644 --- a/src/Parsers/ASTQueryWithOnCluster.cpp +++ b/src/Parsers/ASTQueryWithOnCluster.cpp @@ -5,6 +5,7 @@ #include #include #include +#include namespace DB diff --git a/src/Parsers/ASTQueryWithOutput.h b/src/Parsers/ASTQueryWithOutput.h index 9018d5661d9..92f9331f259 100644 --- a/src/Parsers/ASTQueryWithOutput.h +++ b/src/Parsers/ASTQueryWithOutput.h @@ -1,6 +1,7 @@ #pragma once #include +#include namespace DB diff --git a/src/Parsers/ASTQueryWithTableAndOutput.cpp b/src/Parsers/ASTQueryWithTableAndOutput.cpp index 3a776590f80..d44ba988d7a 100644 --- a/src/Parsers/ASTQueryWithTableAndOutput.cpp +++ b/src/Parsers/ASTQueryWithTableAndOutput.cpp @@ -1,5 +1,6 @@ #include #include +#include namespace DB diff --git a/src/Parsers/ASTRenameQuery.h b/src/Parsers/ASTRenameQuery.h index 951abbe4419..8d300f430b3 100644 --- a/src/Parsers/ASTRenameQuery.h +++ b/src/Parsers/ASTRenameQuery.h @@ -4,6 +4,7 @@ #include #include #include +#include namespace DB diff --git a/src/Parsers/ASTRolesOrUsersSet.cpp b/src/Parsers/ASTRolesOrUsersSet.cpp index a666d8ae1d5..1e7cd79f527 100644 --- a/src/Parsers/ASTRolesOrUsersSet.cpp +++ b/src/Parsers/ASTRolesOrUsersSet.cpp @@ -1,5 +1,6 @@ #include #include +#include namespace DB diff --git a/src/Parsers/ASTRowPolicyName.cpp b/src/Parsers/ASTRowPolicyName.cpp index 5e3c494ccd3..3d1ac5621db 100644 --- a/src/Parsers/ASTRowPolicyName.cpp +++ b/src/Parsers/ASTRowPolicyName.cpp @@ -1,4 +1,5 @@ #include +#include namespace DB diff --git a/src/Parsers/ASTSampleRatio.cpp b/src/Parsers/ASTSampleRatio.cpp index 8c5901d121d..03a4c9adf23 100644 --- a/src/Parsers/ASTSampleRatio.cpp +++ b/src/Parsers/ASTSampleRatio.cpp @@ -1,4 +1,5 @@ #include +#include namespace DB { @@ -34,5 +35,9 @@ String ASTSampleRatio::toString(Rational ratio) return toString(ratio.numerator) + " / " + toString(ratio.denominator); } +void ASTSampleRatio::formatImpl(const IAST::FormatSettings & settings, IAST::FormatState &, IAST::FormatStateStacked) const +{ + settings.ostr << toString(ratio); +} } diff --git a/src/Parsers/ASTSampleRatio.h b/src/Parsers/ASTSampleRatio.h index 787833eb4f3..e8953dec022 100644 --- a/src/Parsers/ASTSampleRatio.h +++ b/src/Parsers/ASTSampleRatio.h @@ -35,10 +35,7 @@ public: static String toString(BigNum num); static String toString(Rational ratio); - void formatImpl(const FormatSettings & settings, FormatState &, FormatStateStacked) const override - { - settings.ostr << toString(ratio); - } + void formatImpl(const FormatSettings & settings, FormatState &, FormatStateStacked) const override; }; } diff --git a/src/Parsers/ASTSelectQuery.cpp b/src/Parsers/ASTSelectQuery.cpp index 499761c4634..915d1f71925 100644 --- a/src/Parsers/ASTSelectQuery.cpp +++ b/src/Parsers/ASTSelectQuery.cpp @@ -6,6 +6,7 @@ #include #include #include +#include namespace DB diff --git a/src/Parsers/ASTSelectWithUnionQuery.cpp b/src/Parsers/ASTSelectWithUnionQuery.cpp index 96cac839c58..9c13a1a4ff3 100644 --- a/src/Parsers/ASTSelectWithUnionQuery.cpp +++ b/src/Parsers/ASTSelectWithUnionQuery.cpp @@ -1,6 +1,7 @@ #include #include #include +#include namespace DB diff --git a/src/Parsers/ASTSetQuery.cpp b/src/Parsers/ASTSetQuery.cpp index 8835d1dc7da..c8a2b3b37e8 100644 --- a/src/Parsers/ASTSetQuery.cpp +++ b/src/Parsers/ASTSetQuery.cpp @@ -1,6 +1,7 @@ #include #include #include +#include namespace DB diff --git a/src/Parsers/ASTSetRoleQuery.cpp b/src/Parsers/ASTSetRoleQuery.cpp index b5e0c05e083..e59e103b774 100644 --- a/src/Parsers/ASTSetRoleQuery.cpp +++ b/src/Parsers/ASTSetRoleQuery.cpp @@ -1,6 +1,7 @@ #include #include #include +#include namespace DB diff --git a/src/Parsers/ASTSettingsProfileElement.cpp b/src/Parsers/ASTSettingsProfileElement.cpp index c0fb2965a2d..2422126219f 100644 --- a/src/Parsers/ASTSettingsProfileElement.cpp +++ b/src/Parsers/ASTSettingsProfileElement.cpp @@ -2,6 +2,7 @@ #include #include #include +#include namespace DB diff --git a/src/Parsers/ASTShowAccessEntitiesQuery.cpp b/src/Parsers/ASTShowAccessEntitiesQuery.cpp index e87baebba33..bacde098640 100644 --- a/src/Parsers/ASTShowAccessEntitiesQuery.cpp +++ b/src/Parsers/ASTShowAccessEntitiesQuery.cpp @@ -1,5 +1,6 @@ #include #include +#include namespace DB diff --git a/src/Parsers/ASTShowCreateAccessEntityQuery.cpp b/src/Parsers/ASTShowCreateAccessEntityQuery.cpp index bc309ab5c44..f870c98071c 100644 --- a/src/Parsers/ASTShowCreateAccessEntityQuery.cpp +++ b/src/Parsers/ASTShowCreateAccessEntityQuery.cpp @@ -1,6 +1,7 @@ #include #include #include +#include namespace DB diff --git a/src/Parsers/ASTShowGrantsQuery.cpp b/src/Parsers/ASTShowGrantsQuery.cpp index 26ae506d7d4..4011cfc522c 100644 --- a/src/Parsers/ASTShowGrantsQuery.cpp +++ b/src/Parsers/ASTShowGrantsQuery.cpp @@ -1,6 +1,7 @@ #include #include #include +#include namespace DB diff --git a/src/Parsers/ASTShowTablesQuery.cpp b/src/Parsers/ASTShowTablesQuery.cpp index 1e8dad13ad3..cd83bae06d9 100644 --- a/src/Parsers/ASTShowTablesQuery.cpp +++ b/src/Parsers/ASTShowTablesQuery.cpp @@ -1,6 +1,7 @@ #include #include #include +#include namespace DB { @@ -21,7 +22,7 @@ void ASTShowTablesQuery::formatLike(const FormatSettings & settings) const << (not_like ? " NOT" : "") << (case_insensitive_like ? " ILIKE " : " LIKE ") << (settings.hilite ? hilite_none : "") - << std::quoted(like, '\''); + << DB::quote << like; } void ASTShowTablesQuery::formatLimit(const FormatSettings & settings, FormatState & state, FormatStateStacked frame) const diff --git a/src/Parsers/ASTSubquery.cpp b/src/Parsers/ASTSubquery.cpp index 55ea89e3f07..bfe413908b9 100644 --- a/src/Parsers/ASTSubquery.cpp +++ b/src/Parsers/ASTSubquery.cpp @@ -1,5 +1,6 @@ #include #include +#include namespace DB { diff --git a/src/Parsers/ASTSystemQuery.cpp b/src/Parsers/ASTSystemQuery.cpp index 9cbb6ae94f6..4ed0ecd3a91 100644 --- a/src/Parsers/ASTSystemQuery.cpp +++ b/src/Parsers/ASTSystemQuery.cpp @@ -1,6 +1,7 @@ #include #include #include +#include namespace DB diff --git a/src/Parsers/ASTTTLElement.cpp b/src/Parsers/ASTTTLElement.cpp index f37631769b8..39283a3168e 100644 --- a/src/Parsers/ASTTTLElement.cpp +++ b/src/Parsers/ASTTTLElement.cpp @@ -2,6 +2,7 @@ #include #include #include +#include namespace DB diff --git a/src/Parsers/ASTTablesInSelectQuery.cpp b/src/Parsers/ASTTablesInSelectQuery.cpp index eb3446ca1c4..8d131a848f7 100644 --- a/src/Parsers/ASTTablesInSelectQuery.cpp +++ b/src/Parsers/ASTTablesInSelectQuery.cpp @@ -2,6 +2,7 @@ #include #include #include +#include namespace DB diff --git a/src/Parsers/ASTUseQuery.h b/src/Parsers/ASTUseQuery.h index 2127bf9f2c0..4e4a13c2a7f 100644 --- a/src/Parsers/ASTUseQuery.h +++ b/src/Parsers/ASTUseQuery.h @@ -2,6 +2,7 @@ #include #include +#include namespace DB diff --git a/src/Parsers/ASTUserNameWithHost.cpp b/src/Parsers/ASTUserNameWithHost.cpp index 13d34b99b3d..b99ea5ab8d4 100644 --- a/src/Parsers/ASTUserNameWithHost.cpp +++ b/src/Parsers/ASTUserNameWithHost.cpp @@ -1,5 +1,6 @@ #include #include +#include namespace DB diff --git a/src/Parsers/ASTWithAlias.cpp b/src/Parsers/ASTWithAlias.cpp index ad93102e1b7..20f647fb575 100644 --- a/src/Parsers/ASTWithAlias.cpp +++ b/src/Parsers/ASTWithAlias.cpp @@ -1,6 +1,6 @@ #include -#include #include +#include namespace DB diff --git a/src/Parsers/ASTWithElement.cpp b/src/Parsers/ASTWithElement.cpp index 9d22286c2fd..00a82b703af 100644 --- a/src/Parsers/ASTWithElement.cpp +++ b/src/Parsers/ASTWithElement.cpp @@ -1,4 +1,5 @@ #include +#include namespace DB { diff --git a/src/Parsers/CommonParsers.cpp b/src/Parsers/CommonParsers.cpp index 47868f5df48..d7a9ed60ac3 100644 --- a/src/Parsers/CommonParsers.cpp +++ b/src/Parsers/CommonParsers.cpp @@ -1,6 +1,7 @@ #include #include #include +#include #include /// strncmp, strncasecmp diff --git a/src/Parsers/DumpASTNode.h b/src/Parsers/DumpASTNode.h index 01447850c74..27ea3b50d62 100644 --- a/src/Parsers/DumpASTNode.h +++ b/src/Parsers/DumpASTNode.h @@ -1,8 +1,8 @@ #pragma once -#include #include #include +#include #include @@ -14,7 +14,7 @@ namespace DB class DumpASTNode { public: - DumpASTNode(const IAST & ast_, std::ostream * ostr_, size_t & depth, const char * label_ = nullptr) + DumpASTNode(const IAST & ast_, WriteBuffer * ostr_, size_t & depth, const char * label_ = nullptr) : ast(ast_), ostr(ostr_), indent(depth), @@ -24,12 +24,12 @@ public: if (!ostr) return; if (label && visit_depth == 0) - (*ostr) << "-- " << label << std::endl; + (*ostr) << "-- " << label << '\n'; ++visit_depth; (*ostr) << String(indent, ' '); printNode(); - (*ostr) << std::endl; + (*ostr) << '\n'; } ~DumpASTNode() @@ -38,7 +38,7 @@ public: return; --visit_depth; if (label && visit_depth == 0) - (*ostr) << "--" << std::endl; + (*ostr) << "--\n"; } template @@ -50,14 +50,14 @@ public: (*ostr) << (str_indent ? String(str_indent) : String(indent, ' ')); (*ostr) << '(' << name << ' ' << value << ')'; if (!str_indent) - (*ostr) << std::endl; + (*ostr) << '\n'; } size_t & getDepth() { return visit_depth; } private: const IAST & ast; - std::ostream * ostr; + WriteBuffer * ostr; size_t indent; size_t & visit_depth; /// shared with children const char * label; @@ -77,7 +77,7 @@ private: } }; -inline void dumpAST(const IAST & ast, std::ostream & ostr, DumpASTNode * parent = nullptr) +inline void dumpAST(const IAST & ast, WriteBuffer & ostr, DumpASTNode * parent = nullptr) { size_t depth = 0; DumpASTNode dump(ast, &ostr, (parent ? parent->getDepth() : depth)); @@ -95,7 +95,6 @@ public: DebugASTLog() : log(nullptr) { - ss.exceptions(std::ios::failbit); if constexpr (_enable) log = &Poco::Logger::get("AST"); } @@ -103,14 +102,14 @@ public: ~DebugASTLog() { if constexpr (_enable) - LOG_DEBUG(log, ss.str()); + LOG_DEBUG(log, buf.str()); } - std::ostream * stream() { return (_enable ? &ss : nullptr); } + WriteBuffer * stream() { return (_enable ? &buf : nullptr); } private: Poco::Logger * log; - std::stringstream ss; + WriteBufferFromOwnString buf; }; diff --git a/src/Parsers/IAST.cpp b/src/Parsers/IAST.cpp index d716a796b77..bc00e2afe77 100644 --- a/src/Parsers/IAST.cpp +++ b/src/Parsers/IAST.cpp @@ -1,4 +1,3 @@ -#include #include #include #include @@ -89,10 +88,9 @@ size_t IAST::checkDepthImpl(size_t max_depth, size_t level) const std::string IAST::formatForErrorMessage() const { - std::stringstream ss; - ss.exceptions(std::ios::failbit); - format(FormatSettings(ss, true /* one line */)); - return ss.str(); + WriteBufferFromOwnString buf; + format(FormatSettings(buf, true /* one line */)); + return buf.str(); } void IAST::cloneChildren() @@ -112,8 +110,6 @@ String IAST::getColumnName() const void IAST::FormatSettings::writeIdentifier(const String & name) const { - WriteBufferFromOStream out(ostr, 32); - switch (identifier_quoting_style) { case IdentifierQuotingStyle::None: @@ -121,36 +117,44 @@ void IAST::FormatSettings::writeIdentifier(const String & name) const if (always_quote_identifiers) throw Exception("Incompatible arguments: always_quote_identifiers = true && identifier_quoting_style == IdentifierQuotingStyle::None", ErrorCodes::BAD_ARGUMENTS); - writeString(name, out); + writeString(name, ostr); break; } case IdentifierQuotingStyle::Backticks: { if (always_quote_identifiers) - writeBackQuotedString(name, out); + writeBackQuotedString(name, ostr); else - writeProbablyBackQuotedString(name, out); + writeProbablyBackQuotedString(name, ostr); break; } case IdentifierQuotingStyle::DoubleQuotes: { if (always_quote_identifiers) - writeDoubleQuotedString(name, out); + writeDoubleQuotedString(name, ostr); else - writeProbablyDoubleQuotedString(name, out); + writeProbablyDoubleQuotedString(name, ostr); break; } case IdentifierQuotingStyle::BackticksMySQL: { if (always_quote_identifiers) - writeBackQuotedStringMySQL(name, out); + writeBackQuotedStringMySQL(name, ostr); else - writeProbablyBackQuotedStringMySQL(name, out); + writeProbablyBackQuotedStringMySQL(name, ostr); break; } } +} - out.next(); +void IAST::dumpTree(WriteBuffer & ostr, size_t indent) const +{ + String indent_str(indent, '-'); + ostr << indent_str << getID() << ", "; + writePointerHex(this, ostr); + writeChar('\n', ostr); + for (const auto & child : children) + child->dumpTree(ostr, indent + 1); } } diff --git a/src/Parsers/IAST.h b/src/Parsers/IAST.h index cc9e593d7cb..01ce4971c45 100644 --- a/src/Parsers/IAST.h +++ b/src/Parsers/IAST.h @@ -5,11 +5,10 @@ #include #include #include +#include #include -#include #include -#include class SipHash; @@ -77,13 +76,7 @@ public: void updateTreeHash(SipHash & hash_state) const; virtual void updateTreeHashImpl(SipHash & hash_state) const; - void dumpTree(std::ostream & ostr, size_t indent = 0) const - { - String indent_str(indent, '-'); - ostr << indent_str << getID() << ", " << this << std::endl; - for (const auto & child : children) - child->dumpTree(ostr, indent + 1); - } + void dumpTree(WriteBuffer & ostr, size_t indent = 0) const; /** Check the depth of the tree. * If max_depth is specified and the depth is greater - throw an exception. @@ -161,7 +154,7 @@ public: /// Format settings. struct FormatSettings { - std::ostream & ostr; + WriteBuffer & ostr; bool hilite = false; bool one_line; bool always_quote_identifiers = false; @@ -169,13 +162,13 @@ public: char nl_or_ws; - FormatSettings(std::ostream & ostr_, bool one_line_) + FormatSettings(WriteBuffer & ostr_, bool one_line_) : ostr(ostr_), one_line(one_line_) { nl_or_ws = one_line ? ' ' : '\n'; } - FormatSettings(std::ostream & ostr_, const FormatSettings & other) + FormatSettings(WriteBuffer & ostr_, const FormatSettings & other) : ostr(ostr_), hilite(other.hilite), one_line(other.one_line), always_quote_identifiers(other.always_quote_identifiers), identifier_quoting_style(other.identifier_quoting_style) { @@ -242,17 +235,17 @@ private: template std::string IAST::formatForErrorMessage(const AstArray & array) { - std::stringstream ss; - ss.exceptions(std::ios::failbit); + WriteBufferFromOwnString buf; for (size_t i = 0; i < array.size(); ++i) { if (i > 0) { - ss << ", "; + const char * delim = ", "; + buf.write(delim, strlen(delim)); } - array[i]->format(IAST::FormatSettings(ss, true /* one line */)); + array[i]->format(IAST::FormatSettings(buf, true /* one line */)); } - return ss.str(); + return buf.str(); } } diff --git a/src/Parsers/MySQL/tests/gtest_create_parser.cpp b/src/Parsers/MySQL/tests/gtest_create_parser.cpp index 1aaba8d67e4..554b3f0a67d 100644 --- a/src/Parsers/MySQL/tests/gtest_create_parser.cpp +++ b/src/Parsers/MySQL/tests/gtest_create_parser.cpp @@ -6,6 +6,7 @@ #include #include #include +#include using namespace DB; using namespace DB::MySQLParser; @@ -37,6 +38,7 @@ TEST(CreateTableParser, SS) ParserCreateQuery p_create_query; String input = "CREATE TABLE `test_table_1` (`a` int DEFAULT NULL, `b` int DEFAULT NULL) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci"; ASTPtr ast = parseQuery(p_create_query, input.data(), input.data() + input.size(), "", 0, 0); - ast->dumpTree(std::cerr); + WriteBufferFromOStream buf(std::cerr, 4096); + ast->dumpTree(buf); } diff --git a/src/Parsers/formatAST.cpp b/src/Parsers/formatAST.cpp index e19dc715d51..fca8ea0aa35 100644 --- a/src/Parsers/formatAST.cpp +++ b/src/Parsers/formatAST.cpp @@ -1,13 +1,12 @@ #include -#include namespace DB { -void formatAST(const IAST & ast, std::ostream & s, bool hilite, bool one_line) +void formatAST(const IAST & ast, WriteBuffer & buf, bool hilite, bool one_line) { - IAST::FormatSettings settings(s, one_line); + IAST::FormatSettings settings(buf, one_line); settings.hilite = hilite; ast.format(settings); @@ -15,10 +14,9 @@ void formatAST(const IAST & ast, std::ostream & s, bool hilite, bool one_line) String serializeAST(const IAST & ast, bool one_line) { - std::stringstream ss; - ss.exceptions(std::ios::failbit); - formatAST(ast, ss, false, one_line); - return ss.str(); + WriteBufferFromOwnString buf; + formatAST(ast, buf, false, one_line); + return buf.str(); } } diff --git a/src/Parsers/formatAST.h b/src/Parsers/formatAST.h index 685c504514e..15381b62028 100644 --- a/src/Parsers/formatAST.h +++ b/src/Parsers/formatAST.h @@ -7,23 +7,25 @@ namespace DB { +class WriteBuffer; + /** Takes a syntax tree and turns it back into text. * In case of INSERT query, the data will be missing. */ -void formatAST(const IAST & ast, std::ostream & s, bool hilite = true, bool one_line = false); +void formatAST(const IAST & ast, WriteBuffer & buf, bool hilite = true, bool one_line = false); String serializeAST(const IAST & ast, bool one_line = true); -inline std::ostream & operator<<(std::ostream & os, const IAST & ast) +inline WriteBuffer & operator<<(WriteBuffer & buf, const IAST & ast) { - formatAST(ast, os, false, true); - return os; + formatAST(ast, buf, false, true); + return buf; } -inline std::ostream & operator<<(std::ostream & os, const ASTPtr & ast) +inline WriteBuffer & operator<<(WriteBuffer & buf, const ASTPtr & ast) { - formatAST(*ast, os, false, true); - return os; + formatAST(*ast, buf, false, true); + return buf; } } diff --git a/src/Parsers/formatSettingName.cpp b/src/Parsers/formatSettingName.cpp index c305496fdb3..351a1eb8ade 100644 --- a/src/Parsers/formatSettingName.cpp +++ b/src/Parsers/formatSettingName.cpp @@ -2,13 +2,13 @@ #include #include #include -#include +#include namespace DB { -void formatSettingName(const String & setting_name, std::ostream & out) +void formatSettingName(const String & setting_name, WriteBuffer & out) { if (isValidIdentifier(setting_name)) { diff --git a/src/Parsers/formatSettingName.h b/src/Parsers/formatSettingName.h index a700d347a5f..ba819ee2b4c 100644 --- a/src/Parsers/formatSettingName.h +++ b/src/Parsers/formatSettingName.h @@ -7,9 +7,11 @@ namespace DB { +class WriteBuffer; + /// Outputs built-in or custom setting's name. /// The function is like backQuoteIfNeed() but didn't quote with backticks /// if the name consists of identifiers joined with dots. -void formatSettingName(const String & setting_name, std::ostream & out); +void formatSettingName(const String & setting_name, WriteBuffer & out); } diff --git a/src/Parsers/iostream_debug_helpers.cpp b/src/Parsers/iostream_debug_helpers.cpp index 66666f0dbfc..b74d337b22d 100644 --- a/src/Parsers/iostream_debug_helpers.cpp +++ b/src/Parsers/iostream_debug_helpers.cpp @@ -3,6 +3,8 @@ #include #include #include +#include +#include namespace DB { @@ -23,9 +25,10 @@ std::ostream & operator<<(std::ostream & stream, const Expected & what) std::ostream & operator<<(std::ostream & stream, const IAST & what) { - stream << "IAST{"; - what.dumpTree(stream); - stream << "}"; + WriteBufferFromOStream buf(stream, 4096); + buf << "IAST{"; + what.dumpTree(buf); + buf << "}"; return stream; } diff --git a/src/Parsers/queryToString.cpp b/src/Parsers/queryToString.cpp index 44ea721485f..9721aa1f128 100644 --- a/src/Parsers/queryToString.cpp +++ b/src/Parsers/queryToString.cpp @@ -1,6 +1,5 @@ #include #include -#include namespace DB { @@ -11,9 +10,6 @@ namespace DB String queryToString(const IAST & query) { - std::ostringstream out; - out.exceptions(std::ios::failbit); - formatAST(query, out, false, true); - return out.str(); + return serializeAST(query); } } diff --git a/src/Parsers/tests/create_parser.cpp b/src/Parsers/tests/create_parser.cpp index fbdc967fa2a..c241b353b4f 100644 --- a/src/Parsers/tests/create_parser.cpp +++ b/src/Parsers/tests/create_parser.cpp @@ -4,6 +4,7 @@ #include #include #include +#include int main(int, char **) @@ -14,7 +15,8 @@ int main(int, char **) ParserCreateQuery parser; ASTPtr ast = parseQuery(parser, input.data(), input.data() + input.size(), "", 0, 0); - formatAST(*ast, std::cerr); + WriteBufferFromOStream out(std::cerr, 4096); + formatAST(*ast, out); std::cerr << std::endl; return 0; diff --git a/src/Parsers/tests/gtest_dictionary_parser.cpp b/src/Parsers/tests/gtest_dictionary_parser.cpp index c418759aa21..b051cedfb23 100644 --- a/src/Parsers/tests/gtest_dictionary_parser.cpp +++ b/src/Parsers/tests/gtest_dictionary_parser.cpp @@ -8,6 +8,7 @@ #include #include #include +#include #include @@ -17,10 +18,9 @@ using namespace DB; static String astToString(IAST * ast) { - std::ostringstream oss; - oss.exceptions(std::ios::failbit); - dumpAST(*ast, oss); - return oss.str(); + WriteBufferFromOwnString buf; + dumpAST(*ast, buf); + return buf.str(); } /// Tests for external dictionaries DDL parser diff --git a/src/Parsers/tests/select_parser.cpp b/src/Parsers/tests/select_parser.cpp index 7711f0d2527..7c18563659d 100644 --- a/src/Parsers/tests/select_parser.cpp +++ b/src/Parsers/tests/select_parser.cpp @@ -3,6 +3,7 @@ #include #include #include +#include int main(int, char **) @@ -25,7 +26,8 @@ try ASTPtr ast = parseQuery(parser, input.data(), input.data() + input.size(), "", 0, 0); std::cout << "Success." << std::endl; - formatAST(*ast, std::cerr); + WriteBufferFromOStream out(std::cerr, 4096); + formatAST(*ast, out); std::cout << std::endl; return 0; diff --git a/src/Parsers/ya.make b/src/Parsers/ya.make index 3fd173d8e18..0bef6699266 100644 --- a/src/Parsers/ya.make +++ b/src/Parsers/ya.make @@ -52,7 +52,6 @@ SRCS( ASTShowAccessEntitiesQuery.cpp ASTShowCreateAccessEntityQuery.cpp ASTShowGrantsQuery.cpp - ASTShowPrivilegesQuery.cpp ASTShowTablesQuery.cpp ASTSubquery.cpp ASTSystemQuery.cpp diff --git a/src/Processors/Formats/Impl/AvroRowInputFormat.cpp b/src/Processors/Formats/Impl/AvroRowInputFormat.cpp index 8a416ade740..787059ddcc8 100644 --- a/src/Processors/Formats/Impl/AvroRowInputFormat.cpp +++ b/src/Processors/Formats/Impl/AvroRowInputFormat.cpp @@ -160,7 +160,7 @@ static void insertNumber(IColumn & column, WhichDataType type, T value) static std::string nodeToJson(avro::NodePtr root_node) { - std::ostringstream ss; + std::ostringstream ss; // STYLE_CHECK_ALLOW_STD_STRING_STREAM ss.exceptions(std::ios::failbit); root_node->printJson(ss, 0); return ss.str(); diff --git a/src/Processors/Formats/Impl/MySQLOutputFormat.cpp b/src/Processors/Formats/Impl/MySQLOutputFormat.cpp index 067c5cb0ab9..f40261b4561 100644 --- a/src/Processors/Formats/Impl/MySQLOutputFormat.cpp +++ b/src/Processors/Formats/Impl/MySQLOutputFormat.cpp @@ -2,8 +2,6 @@ #include #include #include -#include -#include namespace DB { diff --git a/src/Processors/Formats/Impl/PrettyBlockOutputFormat.cpp b/src/Processors/Formats/Impl/PrettyBlockOutputFormat.cpp index 96a458eb49f..8bd4d36532d 100644 --- a/src/Processors/Formats/Impl/PrettyBlockOutputFormat.cpp +++ b/src/Processors/Formats/Impl/PrettyBlockOutputFormat.cpp @@ -5,6 +5,7 @@ #include #include #include +#include #include #include @@ -163,15 +164,10 @@ void PrettyBlockOutputFormat::write(const Chunk & chunk, PortKind port_kind) ascii_grid_symbols; /// Create separators - std::stringstream top_separator; - std::stringstream middle_names_separator; - std::stringstream middle_values_separator; - std::stringstream bottom_separator; - - top_separator.exceptions(std::ios::failbit); - middle_names_separator.exceptions(std::ios::failbit); - middle_values_separator.exceptions(std::ios::failbit); - bottom_separator.exceptions(std::ios::failbit); + WriteBufferFromOwnString top_separator; + WriteBufferFromOwnString middle_names_separator; + WriteBufferFromOwnString middle_values_separator; + WriteBufferFromOwnString bottom_separator; top_separator << grid_symbols.bold_left_top_corner; middle_names_separator << grid_symbols.bold_left_separator; diff --git a/src/Processors/Formats/Impl/PrettyCompactBlockOutputFormat.cpp b/src/Processors/Formats/Impl/PrettyCompactBlockOutputFormat.cpp index 9320b159836..cfa669ae8ad 100644 --- a/src/Processors/Formats/Impl/PrettyCompactBlockOutputFormat.cpp +++ b/src/Processors/Formats/Impl/PrettyCompactBlockOutputFormat.cpp @@ -1,6 +1,7 @@ #include #include #include +#include #include #include @@ -132,22 +133,17 @@ void PrettyCompactBlockOutputFormat::writeBottom(const Widths & max_widths) const GridSymbols & grid_symbols = format_settings.pretty.charset == FormatSettings::Pretty::Charset::UTF8 ? utf8_grid_symbols : ascii_grid_symbols; - /// Create delimiters - std::stringstream bottom_separator; - bottom_separator.exceptions(std::ios::failbit); - - bottom_separator << grid_symbols.left_bottom_corner; + /// Write delimiters + out << grid_symbols.left_bottom_corner; for (size_t i = 0; i < max_widths.size(); ++i) { if (i != 0) - bottom_separator << grid_symbols.bottom_separator; + out << grid_symbols.bottom_separator; for (size_t j = 0; j < max_widths[i] + 2; ++j) - bottom_separator << grid_symbols.dash; + out << grid_symbols.dash; } - bottom_separator << grid_symbols.right_bottom_corner << "\n"; - - writeString(bottom_separator.str(), out); + out << grid_symbols.right_bottom_corner << "\n"; } void PrettyCompactBlockOutputFormat::writeRow( diff --git a/src/Processors/Merges/Algorithms/CollapsingSortedAlgorithm.cpp b/src/Processors/Merges/Algorithms/CollapsingSortedAlgorithm.cpp index e7a7200ac34..8f6888b9a7a 100644 --- a/src/Processors/Merges/Algorithms/CollapsingSortedAlgorithm.cpp +++ b/src/Processors/Merges/Algorithms/CollapsingSortedAlgorithm.cpp @@ -4,6 +4,7 @@ #include #include #include +#include #include @@ -41,8 +42,7 @@ void CollapsingSortedAlgorithm::reportIncorrectData() if (!log) return; - std::stringstream s; - s.exceptions(std::ios::failbit); + WriteBufferFromOwnString s; auto & sort_columns = *last_row.sort_columns; for (size_t i = 0, size = sort_columns.size(); i < size; ++i) { diff --git a/src/Server/HTTPHandler.cpp b/src/Server/HTTPHandler.cpp index 5e03e1d178c..94d66d44af0 100644 --- a/src/Server/HTTPHandler.cpp +++ b/src/Server/HTTPHandler.cpp @@ -351,6 +351,8 @@ void HTTPHandler::processQuery( http_response_compression_method = CompressionMethod::Gzip; else if (std::string::npos != http_response_compression_methods.find("deflate")) http_response_compression_method = CompressionMethod::Zlib; + else if (std::string::npos != http_response_compression_methods.find("xz")) + http_response_compression_method = CompressionMethod::Xz; } bool client_supports_http_compression = http_response_compression_method != CompressionMethod::None; diff --git a/src/Server/ReplicasStatusHandler.cpp b/src/Server/ReplicasStatusHandler.cpp index bf7a3b8ab52..fc79ad9d134 100644 --- a/src/Server/ReplicasStatusHandler.cpp +++ b/src/Server/ReplicasStatusHandler.cpp @@ -36,8 +36,7 @@ void ReplicasStatusHandler::handleRequest(Poco::Net::HTTPServerRequest & request const MergeTreeSettings & settings = context.getReplicatedMergeTreeSettings(); bool ok = true; - std::stringstream message; - message.exceptions(std::ios::failbit); + WriteBufferFromOwnString message; auto databases = DatabaseCatalog::instance().getDatabases(); @@ -83,7 +82,7 @@ void ReplicasStatusHandler::handleRequest(Poco::Net::HTTPServerRequest & request } if (verbose) - response.send() << message.rdbuf(); + response.send() << message.str(); else { const char * data = "Ok.\n"; diff --git a/src/Server/TCPHandler.cpp b/src/Server/TCPHandler.cpp index 4dceb0aa905..f9d21b568b6 100644 --- a/src/Server/TCPHandler.cpp +++ b/src/Server/TCPHandler.cpp @@ -436,13 +436,9 @@ bool TCPHandler::readDataNext(const size_t & poll_interval, const int & receive_ double elapsed = watch.elapsedSeconds(); if (elapsed > receive_timeout) { - std::stringstream ss; - ss.exceptions(std::ios::failbit); - ss << "Timeout exceeded while receiving data from client."; - ss << " Waited for " << static_cast(elapsed) << " seconds,"; - ss << " timeout is " << receive_timeout << " seconds."; - - throw Exception(ss.str(), ErrorCodes::SOCKET_TIMEOUT); + throw Exception(ErrorCodes::SOCKET_TIMEOUT, + "Timeout exceeded while receiving data from client. Waited for {} seconds, timeout is {} seconds.", + static_cast(elapsed), receive_timeout); } } diff --git a/src/Storages/Distributed/DistributedBlockOutputStream.cpp b/src/Storages/Distributed/DistributedBlockOutputStream.cpp index ef7c02163bb..093e5cea8ce 100644 --- a/src/Storages/Distributed/DistributedBlockOutputStream.cpp +++ b/src/Storages/Distributed/DistributedBlockOutputStream.cpp @@ -147,8 +147,7 @@ void DistributedBlockOutputStream::writeAsync(const Block & block) std::string DistributedBlockOutputStream::getCurrentStateDescription() { - std::stringstream buffer; - buffer.exceptions(std::ios::failbit); + WriteBufferFromOwnString buffer; const auto & addresses = cluster->getShardsAddresses(); buffer << "Insertion status:\n"; diff --git a/src/Storages/Kafka/StorageKafka.cpp b/src/Storages/Kafka/StorageKafka.cpp index 03eef37802f..effd2869be2 100644 --- a/src/Storages/Kafka/StorageKafka.cpp +++ b/src/Storages/Kafka/StorageKafka.cpp @@ -246,10 +246,7 @@ Names StorageKafka::parseTopics(String topic_list) String StorageKafka::getDefaultClientId(const StorageID & table_id_) { - std::stringstream ss; - ss.exceptions(std::ios::failbit); - ss << VERSION_NAME << "-" << getFQDNOrHostName() << "-" << table_id_.database_name << "-" << table_id_.table_name; - return ss.str(); + return fmt::format("{}-{}-{}-{}", VERSION_NAME, getFQDNOrHostName(), table_id_.database_name, table_id_.table_name); } @@ -400,10 +397,7 @@ ConsumerBufferPtr StorageKafka::createReadBuffer(const size_t consumer_number) conf.set("group.id", group); if (num_consumers > 1) { - std::stringstream ss; - ss.exceptions(std::ios::failbit); - ss << client_id << "-" << consumer_number; - conf.set("client.id", ss.str()); + conf.set("client.id", fmt::format("{}-{}", client_id, consumer_number)); } else { diff --git a/src/Storages/MergeTree/KeyCondition.cpp b/src/Storages/MergeTree/KeyCondition.cpp index 4caeafc093f..78e43bad73a 100644 --- a/src/Storages/MergeTree/KeyCondition.cpp +++ b/src/Storages/MergeTree/KeyCondition.cpp @@ -17,6 +17,8 @@ #include #include #include +#include +#include #include @@ -33,8 +35,7 @@ namespace ErrorCodes String Range::toString() const { - std::stringstream str; - str.exceptions(std::ios::failbit); + WriteBufferFromOwnString str; if (!left_bounded) str << "(-inf, "; @@ -1559,19 +1560,18 @@ bool KeyCondition::mayBeTrueAfter( String KeyCondition::RPNElement::toString() const { - auto print_wrapped_column = [this](std::ostringstream & ss) + auto print_wrapped_column = [this](WriteBuffer & buf) { for (auto it = monotonic_functions_chain.rbegin(); it != monotonic_functions_chain.rend(); ++it) - ss << (*it)->getName() << "("; + buf << (*it)->getName() << "("; - ss << "column " << key_column; + buf << "column " << key_column; for (auto it = monotonic_functions_chain.rbegin(); it != monotonic_functions_chain.rend(); ++it) - ss << ")"; + buf << ")"; }; - std::ostringstream ss; - ss.exceptions(std::ios::failbit); + WriteBufferFromOwnString buf; switch (function) { case FUNCTION_AND: @@ -1585,24 +1585,24 @@ String KeyCondition::RPNElement::toString() const case FUNCTION_NOT_IN_SET: case FUNCTION_IN_SET: { - ss << "("; - print_wrapped_column(ss); - ss << (function == FUNCTION_IN_SET ? " in " : " notIn "); + buf << "("; + print_wrapped_column(buf); + buf << (function == FUNCTION_IN_SET ? " in " : " notIn "); if (!set_index) - ss << "unknown size set"; + buf << "unknown size set"; else - ss << set_index->size() << "-element set"; - ss << ")"; - return ss.str(); + buf << set_index->size() << "-element set"; + buf << ")"; + return buf.str(); } case FUNCTION_IN_RANGE: case FUNCTION_NOT_IN_RANGE: { - ss << "("; - print_wrapped_column(ss); - ss << (function == FUNCTION_NOT_IN_RANGE ? " not" : "") << " in " << range.toString(); - ss << ")"; - return ss.str(); + buf << "("; + print_wrapped_column(buf); + buf << (function == FUNCTION_NOT_IN_RANGE ? " not" : "") << " in " << range.toString(); + buf << ")"; + return buf.str(); } case ALWAYS_FALSE: return "false"; diff --git a/src/Storages/MergeTree/KeyCondition.h b/src/Storages/MergeTree/KeyCondition.h index 265bc01be49..7c29b1582a6 100644 --- a/src/Storages/MergeTree/KeyCondition.h +++ b/src/Storages/MergeTree/KeyCondition.h @@ -1,6 +1,5 @@ #pragma once -#include #include #include diff --git a/src/Storages/MergeTree/MergeTreeData.cpp b/src/Storages/MergeTree/MergeTreeData.cpp index 175bc72b08d..c30aeae7584 100644 --- a/src/Storages/MergeTree/MergeTreeData.cpp +++ b/src/Storages/MergeTree/MergeTreeData.cpp @@ -3216,13 +3216,12 @@ void MergeTreeData::Transaction::rollbackPartsToTemporaryState() { if (!isEmpty()) { - std::stringstream ss; - ss.exceptions(std::ios::failbit); - ss << " Rollbacking parts state to temporary and removing from working set:"; + WriteBufferFromOwnString buf; + buf << " Rollbacking parts state to temporary and removing from working set:"; for (const auto & part : precommitted_parts) - ss << " " << part->relative_path; - ss << "."; - LOG_DEBUG(data.log, "Undoing transaction.{}", ss.str()); + buf << " " << part->relative_path; + buf << "."; + LOG_DEBUG(data.log, "Undoing transaction.{}", buf.str()); data.removePartsFromWorkingSetImmediatelyAndSetTemporaryState( DataPartsVector(precommitted_parts.begin(), precommitted_parts.end())); @@ -3235,13 +3234,12 @@ void MergeTreeData::Transaction::rollback() { if (!isEmpty()) { - std::stringstream ss; - ss.exceptions(std::ios::failbit); - ss << " Removing parts:"; + WriteBufferFromOwnString buf; + buf << " Removing parts:"; for (const auto & part : precommitted_parts) - ss << " " << part->relative_path; - ss << "."; - LOG_DEBUG(data.log, "Undoing transaction.{}", ss.str()); + buf << " " << part->relative_path; + buf << "."; + LOG_DEBUG(data.log, "Undoing transaction.{}", buf.str()); data.removePartsFromWorkingSet( DataPartsVector(precommitted_parts.begin(), precommitted_parts.end()), @@ -3771,15 +3769,15 @@ bool MergeTreeData::canUsePolymorphicParts(const MergeTreeSettings & settings, S if (out_reason && (settings.min_rows_for_wide_part != 0 || settings.min_bytes_for_wide_part != 0 || settings.min_rows_for_compact_part != 0 || settings.min_bytes_for_compact_part != 0)) { - std::ostringstream message; - message.exceptions(std::ios::failbit); - message << "Table can't create parts with adaptive granularity, but settings" - << " min_rows_for_wide_part = " << settings.min_rows_for_wide_part - << ", min_bytes_for_wide_part = " << settings.min_bytes_for_wide_part - << ", min_rows_for_compact_part = " << settings.min_rows_for_compact_part - << ", min_bytes_for_compact_part = " << settings.min_bytes_for_compact_part - << ". Parts with non-adaptive granularity can be stored only in Wide (default) format."; - *out_reason = message.str(); + *out_reason = fmt::format( + "Table can't create parts with adaptive granularity, but settings" + " min_rows_for_wide_part = {}" + ", min_bytes_for_wide_part = {}" + ", min_rows_for_compact_part = {}" + ", min_bytes_for_compact_part = {}" + ". Parts with non-adaptive granularity can be stored only in Wide (default) format.", + settings.min_rows_for_wide_part, settings.min_bytes_for_wide_part, + settings.min_rows_for_compact_part, settings.min_bytes_for_compact_part); } return false; diff --git a/src/Storages/MergeTree/MergeTreeDataPartChecksum.cpp b/src/Storages/MergeTree/MergeTreeDataPartChecksum.cpp index 58bdbcdcdcd..dd141a68248 100644 --- a/src/Storages/MergeTree/MergeTreeDataPartChecksum.cpp +++ b/src/Storages/MergeTree/MergeTreeDataPartChecksum.cpp @@ -442,12 +442,9 @@ void MinimalisticDataPartChecksums::checkEqualImpl(const MinimalisticDataPartChe { if (num_compressed_files != rhs.num_compressed_files || num_uncompressed_files != rhs.num_uncompressed_files) { - std::stringstream error_msg; - error_msg.exceptions(std::ios::failbit); - error_msg << "Different number of files: " << rhs.num_compressed_files << " compressed (expected " << num_compressed_files << ")" - << " and " << rhs.num_uncompressed_files << " uncompressed ones (expected " << num_uncompressed_files << ")"; - - throw Exception(error_msg.str(), ErrorCodes::CHECKSUM_DOESNT_MATCH); + throw Exception(ErrorCodes::CHECKSUM_DOESNT_MATCH, + "Different number of files: {} compressed (expected {}) and {} uncompressed ones (expected {})", + rhs.num_compressed_files, num_compressed_files, rhs.num_uncompressed_files, num_uncompressed_files); } Strings errors; diff --git a/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp b/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp index 7f031771373..5e3f76a19f4 100644 --- a/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp +++ b/src/Storages/MergeTree/MergeTreeDataSelectExecutor.cpp @@ -213,14 +213,8 @@ QueryPlanPtr MergeTreeDataSelectExecutor::readFromParts( if (settings.force_primary_key && key_condition.alwaysUnknownOrTrue()) { - std::stringstream exception_message; - exception_message.exceptions(std::ios::failbit); - exception_message << "Primary key ("; - for (size_t i = 0, size = primary_key_columns.size(); i < size; ++i) - exception_message << (i == 0 ? "" : ", ") << primary_key_columns[i]; - exception_message << ") is not used and setting 'force_primary_key' is set."; - - throw Exception(exception_message.str(), ErrorCodes::INDEX_NOT_USED); + throw Exception(ErrorCodes::INDEX_NOT_USED, "Primary key ({}) is not used and setting 'force_primary_key' is set.", + boost::algorithm::join(primary_key_columns, ", ")); } std::optional minmax_idx_condition; diff --git a/src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp b/src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp index 597ff5e8fee..9263dab638a 100644 --- a/src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp +++ b/src/Storages/MergeTree/ReplicatedMergeTreeQueue.cpp @@ -941,8 +941,7 @@ size_t ReplicatedMergeTreeQueue::getConflictsCountForRange( if (out_description) { - std::stringstream ss; - ss.exceptions(std::ios::failbit); + WriteBufferFromOwnString ss; ss << "Can't execute command for range " << range.getPartName() << " (entry " << entry.znode_name << "). "; ss << "There are " << conflicts.size() << " currently executing entries blocking it: "; for (const auto & conflict : conflicts) @@ -1693,13 +1692,12 @@ std::vector ReplicatedMergeTreeQueue::getMutationsStatu for (const MutationCommand & command : entry.commands) { - std::stringstream ss; - ss.exceptions(std::ios::failbit); - formatAST(*command.ast, ss, false, true); + WriteBufferFromOwnString buf; + formatAST(*command.ast, buf, false, true); result.push_back(MergeTreeMutationStatus { entry.znode_name, - ss.str(), + buf.str(), entry.create_time, entry.block_numbers, parts_to_mutate, diff --git a/src/Storages/MergeTree/ReplicatedMergeTreeTableMetadata.cpp b/src/Storages/MergeTree/ReplicatedMergeTreeTableMetadata.cpp index 48f05b50675..d06706f9109 100644 --- a/src/Storages/MergeTree/ReplicatedMergeTreeTableMetadata.cpp +++ b/src/Storages/MergeTree/ReplicatedMergeTreeTableMetadata.cpp @@ -18,10 +18,9 @@ static String formattedAST(const ASTPtr & ast) { if (!ast) return ""; - std::stringstream ss; - ss.exceptions(std::ios::failbit); - formatAST(*ast, ss, false, true); - return ss.str(); + WriteBufferFromOwnString buf; + formatAST(*ast, buf, false, true); + return buf.str(); } ReplicatedMergeTreeTableMetadata::ReplicatedMergeTreeTableMetadata(const MergeTreeData & data, const StorageMetadataPtr & metadata_snapshot) diff --git a/src/Storages/MutationCommands.cpp b/src/Storages/MutationCommands.cpp index bb22a1d0395..daa7665403f 100644 --- a/src/Storages/MutationCommands.cpp +++ b/src/Storages/MutationCommands.cpp @@ -131,10 +131,9 @@ std::shared_ptr MutationCommands::ast() const void MutationCommands::writeText(WriteBuffer & out) const { - std::stringstream commands_ss; - commands_ss.exceptions(std::ios::failbit); - formatAST(*ast(), commands_ss, /* hilite = */ false, /* one_line = */ true); - out << escape << commands_ss.str(); + WriteBufferFromOwnString commands_buf; + formatAST(*ast(), commands_buf, /* hilite = */ false, /* one_line = */ true); + out << escape << commands_buf.str(); } void MutationCommands::readText(ReadBuffer & in) diff --git a/src/Storages/RabbitMQ/StorageRabbitMQ.cpp b/src/Storages/RabbitMQ/StorageRabbitMQ.cpp index f8128e6223a..f9e3c0558d3 100644 --- a/src/Storages/RabbitMQ/StorageRabbitMQ.cpp +++ b/src/Storages/RabbitMQ/StorageRabbitMQ.cpp @@ -186,15 +186,10 @@ AMQP::ExchangeType StorageRabbitMQ::defineExchangeType(String exchange_type_) String StorageRabbitMQ::getTableBasedName(String name, const StorageID & table_id) { - std::stringstream ss; - ss.exceptions(std::ios::failbit); - if (name.empty()) - ss << table_id.database_name << "_" << table_id.table_name; + return fmt::format("{}_{}", table_id.database_name, table_id.table_name); else - ss << name << "_" << table_id.database_name << "_" << table_id.table_name; - - return ss.str(); + return fmt::format("{}_{}_{}", name, table_id.database_name, table_id.table_name); } diff --git a/src/Storages/StorageDictionary.cpp b/src/Storages/StorageDictionary.cpp index ee08dd5a824..f81bcbc42d0 100644 --- a/src/Storages/StorageDictionary.cpp +++ b/src/Storages/StorageDictionary.cpp @@ -9,7 +9,7 @@ #include #include #include -#include +#include namespace DB @@ -32,12 +32,9 @@ namespace { if (names_and_types_set.find(column) == names_and_types_set.end()) { - std::string message = "Not found column "; - message += column.name + " " + column.type->getName(); - message += " in dictionary " + backQuote(dictionary_name) + ". "; - message += "There are only columns "; - message += StorageDictionary::generateNamesAndTypesDescription(dictionary_names_and_types); - throw Exception(message, ErrorCodes::THERE_IS_NO_COLUMN); + throw Exception(ErrorCodes::THERE_IS_NO_COLUMN, "Not found column {} {} in dictionary {}. There are only columns {}", + column.name, column.type->getName(), backQuote(dictionary_name), + StorageDictionary::generateNamesAndTypesDescription(dictionary_names_and_types)); } } } @@ -81,8 +78,7 @@ NamesAndTypesList StorageDictionary::getNamesAndTypes(const DictionaryStructure String StorageDictionary::generateNamesAndTypesDescription(const NamesAndTypesList & list) { - std::stringstream ss; - ss.exceptions(std::ios::failbit); + WriteBufferFromOwnString ss; bool first = true; for (const auto & name_and_type : list) { diff --git a/src/Storages/StorageDistributed.cpp b/src/Storages/StorageDistributed.cpp index 237cd5c4017..0e37a16416d 100644 --- a/src/Storages/StorageDistributed.cpp +++ b/src/Storages/StorageDistributed.cpp @@ -49,6 +49,8 @@ #include #include +#include +#include #include @@ -175,19 +177,18 @@ UInt64 getMaximumFileNumber(const std::string & dir_path) std::string makeFormattedListOfShards(const ClusterPtr & cluster) { - std::ostringstream os; - os.exceptions(std::ios::failbit); + WriteBufferFromOwnString buf; bool head = true; - os << "["; + buf << "["; for (const auto & shard_info : cluster->getShardsInfo()) { - (head ? os : os << ", ") << shard_info.shard_num; + (head ? buf : buf << ", ") << shard_info.shard_num; head = false; } - os << "]"; + buf << "]"; - return os.str(); + return buf.str(); } ExpressionActionsPtr buildShardingKeyExpression(const ASTPtr & sharding_key, const Context & context, const NamesAndTypesList & columns, bool project) @@ -760,8 +761,7 @@ ClusterPtr StorageDistributed::getOptimizedCluster(const Context & context, cons UInt64 force = settings.force_optimize_skip_unused_shards; if (force) { - std::stringstream exception_message; - exception_message.exceptions(std::ios::failbit); + WriteBufferFromOwnString exception_message; if (!has_sharding_key) exception_message << "No sharding key"; else if (!sharding_key_is_usable) diff --git a/src/Storages/StorageInMemoryMetadata.cpp b/src/Storages/StorageInMemoryMetadata.cpp index 5cc435f91fa..a4500e2aa7b 100644 --- a/src/Storages/StorageInMemoryMetadata.cpp +++ b/src/Storages/StorageInMemoryMetadata.cpp @@ -4,6 +4,7 @@ #include #include #include +#include namespace DB @@ -427,8 +428,7 @@ namespace String listOfColumns(const NamesAndTypesList & available_columns) { - std::stringstream ss; - ss.exceptions(std::ios::failbit); + WriteBufferFromOwnString ss; for (auto it = available_columns.begin(); it != available_columns.end(); ++it) { if (it != available_columns.begin()) diff --git a/src/Storages/StorageMergeTree.cpp b/src/Storages/StorageMergeTree.cpp index 925f510f63a..3fb6f0cb1e8 100644 --- a/src/Storages/StorageMergeTree.cpp +++ b/src/Storages/StorageMergeTree.cpp @@ -554,13 +554,12 @@ std::vector StorageMergeTree::getMutationsStatus() cons for (const MutationCommand & command : entry.commands) { - std::stringstream ss; - ss.exceptions(std::ios::failbit); - formatAST(*command.ast, ss, false, true); + WriteBufferFromOwnString buf; + formatAST(*command.ast, buf, false, true); result.push_back(MergeTreeMutationStatus { entry.file_name, - ss.str(), + buf.str(), entry.create_time, block_numbers_map, parts_to_do_names, @@ -1030,17 +1029,13 @@ bool StorageMergeTree::optimize( { if (!merge(true, partition_id, true, deduplicate, &disable_reason)) { - std::stringstream message; - message.exceptions(std::ios::failbit); - message << "Cannot OPTIMIZE table"; - if (!disable_reason.empty()) - message << ": " << disable_reason; - else - message << " by some reason."; - LOG_INFO(log, message.str()); + constexpr const char * message = "Cannot OPTIMIZE table: {}"; + if (disable_reason.empty()) + disable_reason = "unknown reason"; + LOG_INFO(log, message, disable_reason); if (context.getSettingsRef().optimize_throw_if_noop) - throw Exception(message.str(), ErrorCodes::CANNOT_ASSIGN_OPTIMIZE); + throw Exception(ErrorCodes::CANNOT_ASSIGN_OPTIMIZE, message, disable_reason); return false; } } @@ -1053,17 +1048,13 @@ bool StorageMergeTree::optimize( if (!merge(true, partition_id, final, deduplicate, &disable_reason)) { - std::stringstream message; - message.exceptions(std::ios::failbit); - message << "Cannot OPTIMIZE table"; - if (!disable_reason.empty()) - message << ": " << disable_reason; - else - message << " by some reason."; - LOG_INFO(log, message.str()); + constexpr const char * message = "Cannot OPTIMIZE table: {}"; + if (disable_reason.empty()) + disable_reason = "unknown reason"; + LOG_INFO(log, message, disable_reason); if (context.getSettingsRef().optimize_throw_if_noop) - throw Exception(message.str(), ErrorCodes::CANNOT_ASSIGN_OPTIMIZE); + throw Exception(ErrorCodes::CANNOT_ASSIGN_OPTIMIZE, message, disable_reason); return false; } } diff --git a/src/Storages/StorageReplicatedMergeTree.cpp b/src/Storages/StorageReplicatedMergeTree.cpp index e7264610e3e..93febf919c9 100644 --- a/src/Storages/StorageReplicatedMergeTree.cpp +++ b/src/Storages/StorageReplicatedMergeTree.cpp @@ -461,16 +461,9 @@ void StorageReplicatedMergeTree::waitMutationToFinishOnReplicas( if (!inactive_replicas.empty()) { - std::stringstream exception_message; - exception_message.exceptions(std::ios::failbit); - exception_message << "Mutation is not finished because"; - - if (!inactive_replicas.empty()) - exception_message << " some replicas are inactive right now: " << boost::algorithm::join(inactive_replicas, ", "); - - exception_message << ". Mutation will be done asynchronously"; - - throw Exception(exception_message.str(), ErrorCodes::UNFINISHED); + throw Exception(ErrorCodes::UNFINISHED, + "Mutation is not finished because some replicas are inactive right now: {}. Mutation will be done asynchronously", + boost::algorithm::join(inactive_replicas, ", ")); } } @@ -1019,13 +1012,6 @@ void StorageReplicatedMergeTree::checkParts(bool skip_sanity_checks) for (const String & name : parts_to_fetch) parts_to_fetch_blocks += get_blocks_count_in_data_part(name); - std::stringstream sanity_report; - sanity_report.exceptions(std::ios::failbit); - sanity_report << "There are " - << unexpected_parts.size() << " unexpected parts with " << unexpected_parts_rows << " rows (" - << unexpected_parts_nonnew << " of them is not just-written with " << unexpected_parts_rows << " rows), " - << parts_to_fetch.size() << " missing parts (with " << parts_to_fetch_blocks << " blocks)."; - /** We can automatically synchronize data, * if the ratio of the total number of errors to the total number of parts (minimum - on the local filesystem or in ZK) * is no more than some threshold (for example 50%). @@ -1042,20 +1028,26 @@ void StorageReplicatedMergeTree::checkParts(bool skip_sanity_checks) const auto storage_settings_ptr = getSettings(); bool insane = unexpected_parts_rows > total_rows_on_filesystem * storage_settings_ptr->replicated_max_ratio_of_wrong_parts; + constexpr const char * sanity_report_fmt = "The local set of parts of table {} doesn't look like the set of parts in ZooKeeper: " + "{} rows of {} total rows in filesystem are suspicious. " + "There are {} unexpected parts with {} rows ({} of them is not just-written with {} rows), " + "{} missing parts (with {} blocks)."; + if (insane && !skip_sanity_checks) { - std::stringstream why; - why.exceptions(std::ios::failbit); - why << "The local set of parts of table " << getStorageID().getNameForLogs() << " doesn't look like the set of parts " - << "in ZooKeeper: " - << formatReadableQuantity(unexpected_parts_rows) << " rows of " << formatReadableQuantity(total_rows_on_filesystem) - << " total rows in filesystem are suspicious."; - - throw Exception(why.str() + " " + sanity_report.str(), ErrorCodes::TOO_MANY_UNEXPECTED_DATA_PARTS); + throw Exception(ErrorCodes::TOO_MANY_UNEXPECTED_DATA_PARTS, sanity_report_fmt, getStorageID().getNameForLogs(), + formatReadableQuantity(unexpected_parts_rows), formatReadableQuantity(total_rows_on_filesystem), + unexpected_parts.size(), unexpected_parts_rows, unexpected_parts_nonnew, unexpected_parts_nonnew_rows, + parts_to_fetch.size(), parts_to_fetch_blocks); } if (unexpected_parts_nonnew_rows > 0) - LOG_WARNING(log, sanity_report.str()); + { + LOG_WARNING(log, sanity_report_fmt, getStorageID().getNameForLogs(), + formatReadableQuantity(unexpected_parts_rows), formatReadableQuantity(total_rows_on_filesystem), + unexpected_parts.size(), unexpected_parts_rows, unexpected_parts_nonnew, unexpected_parts_nonnew_rows, + parts_to_fetch.size(), parts_to_fetch_blocks); + } /// Add to the queue jobs to pick up the missing parts from other replicas and remove from ZK the information that we have them. std::vector> exists_futures; @@ -1344,15 +1336,7 @@ bool StorageReplicatedMergeTree::executeLogEntry(LogEntry & entry) bool StorageReplicatedMergeTree::tryExecuteMerge(const LogEntry & entry) { - // Log source part names just in case - { - std::stringstream source_parts_msg; - source_parts_msg.exceptions(std::ios::failbit); - for (auto i : ext::range(0, entry.source_parts.size())) - source_parts_msg << (i != 0 ? ", " : "") << entry.source_parts[i]; - - LOG_TRACE(log, "Executing log entry to merge parts {} to {}", source_parts_msg.str(), entry.new_part_name); - } + LOG_TRACE(log, "Executing log entry to merge parts {} to {}", boost::algorithm::join(entry.source_parts, ", "), entry.new_part_name); const auto storage_settings_ptr = getSettings(); @@ -3860,13 +3844,11 @@ bool StorageReplicatedMergeTree::optimize( if (!selected) { - std::stringstream message; - message.exceptions(std::ios::failbit); - message << "Cannot select parts for optimization"; - if (!disable_reason.empty()) - message << ": " << disable_reason; - LOG_INFO(log, message.str()); - return handle_noop(message.str()); + constexpr const char * message_fmt = "Cannot select parts for optimization: {}"; + if (disable_reason.empty()) + disable_reason = "unknown reason"; + LOG_INFO(log, message_fmt, disable_reason); + return handle_noop(fmt::format(message_fmt, disable_reason)); } ReplicatedMergeTreeLogEntryData merge_entry; diff --git a/src/Storages/StorageS3.cpp b/src/Storages/StorageS3.cpp index ce9ebbd53b3..e5c16dad958 100644 --- a/src/Storages/StorageS3.cpp +++ b/src/Storages/StorageS3.cpp @@ -35,8 +35,6 @@ #include #include -#include - namespace DB { @@ -254,18 +252,14 @@ Strings listFilesWithRegexpMatching(Aws::S3::S3Client & client, const S3::URI & outcome = client.ListObjectsV2(request); if (!outcome.IsSuccess()) { - std::ostringstream message; - message.exceptions(std::ios::failbit); - message << "Could not list objects in bucket " << quoteString(request.GetBucket()) - << " with prefix " << quoteString(request.GetPrefix()); - if (page > 1) - message << ", page " << std::to_string(page); + throw Exception(ErrorCodes::S3_ERROR, "Could not list objects in bucket {} with prefix {}, page {}, S3 exception: {}, message: {}", + quoteString(request.GetBucket()), quoteString(request.GetPrefix()), page, + backQuote(outcome.GetError().GetExceptionName()), quoteString(outcome.GetError().GetMessage())); - message << ", S3 exception: " + backQuote(outcome.GetError().GetExceptionName()) - << ", message: " + quoteString(outcome.GetError().GetMessage()); - - throw Exception(message.str(), ErrorCodes::S3_ERROR); + throw Exception(ErrorCodes::S3_ERROR, "Could not list objects in bucket {} with prefix {}, S3 exception: {}, message: {}", + quoteString(request.GetBucket()), quoteString(request.GetPrefix()), + backQuote(outcome.GetError().GetExceptionName()), quoteString(outcome.GetError().GetMessage())); } for (const auto & row : outcome.GetResult().GetContents()) diff --git a/src/Storages/System/StorageSystemUsers.cpp b/src/Storages/System/StorageSystemUsers.cpp index 70c67683b25..675fee84746 100644 --- a/src/Storages/System/StorageSystemUsers.cpp +++ b/src/Storages/System/StorageSystemUsers.cpp @@ -96,7 +96,7 @@ void StorageSystemUsers::fillData(MutableColumns & res_columns, const Context & auth_params_json.set("server", authentication.getServerName()); - std::ostringstream oss; + std::ostringstream oss; // STYLE_CHECK_ALLOW_STD_STRING_STREAM oss.exceptions(std::ios::failbit); Poco::JSON::Stringifier::stringify(auth_params_json, oss); const auto str = oss.str(); diff --git a/src/Storages/tests/gtest_storage_log.cpp b/src/Storages/tests/gtest_storage_log.cpp index 821fbc4b279..0fb418e8413 100644 --- a/src/Storages/tests/gtest_storage_log.cpp +++ b/src/Storages/tests/gtest_storage_log.cpp @@ -133,16 +133,14 @@ std::string readData(DB::StoragePtr & table, const DB::Context & context) tryRegisterFormats(); - std::ostringstream ss; - ss.exceptions(std::ios::failbit); - WriteBufferFromOStream out_buf(ss); + WriteBufferFromOwnString out_buf; BlockOutputStreamPtr output = FormatFactory::instance().getOutput("Values", out_buf, sample, context); copyData(*in, *output); output->flush(); - return ss.str(); + return out_buf.str(); } TYPED_TEST(StorageLogTest, testReadWrite) diff --git a/src/Storages/transformQueryForExternalDatabase.cpp b/src/Storages/transformQueryForExternalDatabase.cpp index 3148ab1112a..f35fb1c8a34 100644 --- a/src/Storages/transformQueryForExternalDatabase.cpp +++ b/src/Storages/transformQueryForExternalDatabase.cpp @@ -1,4 +1,3 @@ -#include #include #include #include @@ -220,8 +219,7 @@ String transformQueryForExternalDatabase( ASTPtr select_ptr = select; dropAliases(select_ptr); - std::stringstream out; - out.exceptions(std::ios::failbit); + WriteBufferFromOwnString out; IAST::FormatSettings settings(out, true); settings.identifier_quoting_style = identifier_quoting_style; settings.always_quote_identifiers = identifier_quoting_style != IdentifierQuotingStyle::None; diff --git a/src/TableFunctions/TableFunctionRemote.cpp b/src/TableFunctions/TableFunctionRemote.cpp index 22a07a4d284..a031490b88b 100644 --- a/src/TableFunctions/TableFunctionRemote.cpp +++ b/src/TableFunctions/TableFunctionRemote.cpp @@ -241,13 +241,9 @@ TableFunctionRemote::TableFunctionRemote(const std::string & name_, bool secure_ : name{name_}, secure{secure_} { is_cluster_function = (name == "cluster" || name == "clusterAllReplicas"); - - std::stringstream ss; - ss.exceptions(std::ios::failbit); - ss << "Table function '" << name + "' requires from 2 to " << (is_cluster_function ? 3 : 5) << " parameters" - << ": , , " - << (is_cluster_function ? "" : ", [username, [password]]."); - help_message = ss.str(); + help_message = fmt::format("Table function '{}' requires from 2 to {} parameters: " + ", , {}", + name, is_cluster_function ? 3 : 5, is_cluster_function ? "" : ", [username, [password]]."); } diff --git a/src/Parsers/ASTShowPrivilegesQuery.cpp b/tests/integration/test_reload_auxiliary_zookeepers/__init__.py similarity index 100% rename from src/Parsers/ASTShowPrivilegesQuery.cpp rename to tests/integration/test_reload_auxiliary_zookeepers/__init__.py diff --git a/tests/integration/test_reload_auxiliary_zookeepers/configs/config.xml b/tests/integration/test_reload_auxiliary_zookeepers/configs/config.xml new file mode 100644 index 00000000000..b5e5495c096 --- /dev/null +++ b/tests/integration/test_reload_auxiliary_zookeepers/configs/config.xml @@ -0,0 +1,31 @@ + + + + trace + /var/log/clickhouse-server/clickhouse-server.log + /var/log/clickhouse-server/clickhouse-server.err.log + 1000M + 10 + + + 9000 + 127.0.0.1 + + + + true + none + + AcceptCertificateHandler + + + + + 500 + 5368709120 + ./clickhouse/ + users.xml + + 1 + 1 + diff --git a/tests/integration/test_reload_auxiliary_zookeepers/configs/users.xml b/tests/integration/test_reload_auxiliary_zookeepers/configs/users.xml new file mode 100644 index 00000000000..6061af8e33d --- /dev/null +++ b/tests/integration/test_reload_auxiliary_zookeepers/configs/users.xml @@ -0,0 +1,23 @@ + + + + + + + + + + + + ::/0 + + default + default + + + + + + + + diff --git a/tests/integration/test_reload_auxiliary_zookeepers/configs/zookeeper.xml b/tests/integration/test_reload_auxiliary_zookeepers/configs/zookeeper.xml new file mode 100644 index 00000000000..8157cc1d30b --- /dev/null +++ b/tests/integration/test_reload_auxiliary_zookeepers/configs/zookeeper.xml @@ -0,0 +1,17 @@ + + + + zoo1 + 2181 + + + zoo2 + 2181 + + + zoo3 + 2181 + + 2000 + + diff --git a/tests/integration/test_reload_auxiliary_zookeepers/test.py b/tests/integration/test_reload_auxiliary_zookeepers/test.py new file mode 100644 index 00000000000..92c66c890fc --- /dev/null +++ b/tests/integration/test_reload_auxiliary_zookeepers/test.py @@ -0,0 +1,89 @@ +import time +import pytest +import os + +from helpers.cluster import ClickHouseCluster +from helpers.client import QueryRuntimeException +from helpers.test_tools import assert_eq_with_retry + +cluster = ClickHouseCluster(__file__, zookeeper_config_path="configs/zookeeper.xml") +node = cluster.add_instance("node", with_zookeeper=True) + + +@pytest.fixture(scope="module") +def start_cluster(): + try: + cluster.start() + yield cluster + finally: + cluster.shutdown() + + +def test_reload_auxiliary_zookeepers(start_cluster): + + node.query( + "CREATE TABLE simple (date Date, id UInt32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/0/simple', 'node') ORDER BY tuple() PARTITION BY date;" + ) + node.query("INSERT INTO simple VALUES ('2020-08-27', 1)") + + node.query( + "CREATE TABLE simple2 (date Date, id UInt32) ENGINE = ReplicatedMergeTree('/clickhouse/tables/1/simple', 'node') ORDER BY tuple() PARTITION BY date;" + ) + + # Add an auxiliary zookeeper + new_config = """ + + + zoo1 + 2181 + + + zoo2 + 2181 + + + zoo3 + 2181 + + 2000 + + + + + zoo1 + 2181 + + + zoo2 + 2181 + + + +""" + node.replace_config("/etc/clickhouse-server/conf.d/zookeeper.xml", new_config) + + # Hopefully it has finished the configuration reload + time.sleep(2) + + node.query( + "ALTER TABLE simple2 FETCH PARTITION '2020-08-27' FROM 'zookeeper2:/clickhouse/tables/0/simple';" + ) + node.query("ALTER TABLE simple2 ATTACH PARTITION '2020-08-27';") + assert node.query("SELECT id FROM simple2").strip() == "1" + + new_config = """ + + + zoo2 + 2181 + + 2000 + +""" + node.replace_config("/etc/clickhouse-server/conf.d/zookeeper.xml", new_config) + time.sleep(2) + with pytest.raises(QueryRuntimeException): + node.query( + "ALTER TABLE simple2 FETCH PARTITION '2020-08-27' FROM 'zookeeper2:/clickhouse/tables/0/simple';" + ) + assert node.query("SELECT id FROM simple2").strip() == "1" diff --git a/tests/integration/test_storage_s3/test.py b/tests/integration/test_storage_s3/test.py index 42dbeda4717..4a2cd77e233 100644 --- a/tests/integration/test_storage_s3/test.py +++ b/tests/integration/test_storage_s3/test.py @@ -136,6 +136,40 @@ def test_put(cluster, maybe_auth, positive): assert values_csv == get_s3_file_content(cluster, bucket, filename) +# Test put no data to S3. +@pytest.mark.parametrize("auth", [ + "'minio','minio123'," +]) +def test_empty_put(cluster, auth): + # type: (ClickHouseCluster) -> None + + bucket = cluster.minio_bucket + instance = cluster.instances["dummy"] # type: ClickHouseInstance + table_format = "column1 UInt32, column2 UInt32, column3 UInt32" + + create_empty_table_query = """ + CREATE TABLE empty_table ( + {} + ) ENGINE = Null() + """.format(table_format) + + run_query(instance, create_empty_table_query) + + filename = "empty_put_test.csv" + put_query = "insert into table function s3('http://{}:{}/{}/{}', {}'CSV', '{}') select * from empty_table".format( + cluster.minio_host, cluster.minio_port, bucket, filename, auth, table_format) + + run_query(instance, put_query) + + try: + run_query(instance, "select count(*) from s3('http://{}:{}/{}/{}', {}'CSV', '{}')".format( + cluster.minio_host, cluster.minio_port, bucket, filename, auth, table_format)) + + assert False, "Query should be failed." + except helpers.client.QueryRuntimeException as e: + assert str(e).find("The specified key does not exist") != 0 + + # Test put values in CSV format. @pytest.mark.parametrize("maybe_auth,positive", [ ("", True), diff --git a/tests/performance/push_down_limit.xml b/tests/performance/push_down_limit.xml index 6ae63b54ec6..0611216410d 100644 --- a/tests/performance/push_down_limit.xml +++ b/tests/performance/push_down_limit.xml @@ -4,5 +4,5 @@ select number from (select number from numbers(1500000000) order by -number) limit 10 select number from (select number from numbers_mt(1500000000) order by -number) limit 10 - select number from numbers_view limit 100 + select number from numbers_view limit 100 diff --git a/tests/performance/quantile_merge.xml b/tests/performance/quantile_merge.xml index 0ddb688d8eb..ad0f0571682 100644 --- a/tests/performance/quantile_merge.xml +++ b/tests/performance/quantile_merge.xml @@ -1,3 +1,3 @@ - SELECT quantileMerge(arrayJoin(arrayMap(x -> state, range(500000)))) FROM (SELECT quantileState(rand()) AS state FROM numbers(10000)) + SELECT quantileMerge(arrayJoin(arrayMap(x -> state, range(500000)))) FROM (SELECT quantileState(rand()) AS state FROM numbers(10000)) diff --git a/tests/queries/0_stateless/00039_inserts_through_http.sh b/tests/queries/0_stateless/00039_inserts_through_http.sh index 6b04437dcd6..35abcd166d7 100755 --- a/tests/queries/0_stateless/00039_inserts_through_http.sh +++ b/tests/queries/0_stateless/00039_inserts_through_http.sh @@ -10,3 +10,5 @@ for string_size in 1 10 100 1000 10000 100000 1000000; do LC_ALL=C perl -we 'for my $letter ("a" .. "z") { print(($letter x '$string_size') . "\n") }' | ${CLICKHOUSE_CURL} -sSg "${CLICKHOUSE_URL}&query=INSERT+INTO+long_insert+FORMAT+TabSeparated" --data-binary @- echo 'SELECT substring(a, 1, 1) AS c, length(a) AS l FROM long_insert ORDER BY c, l' | ${CLICKHOUSE_CURL} -sSg "${CLICKHOUSE_URL}" -d @- done + +echo 'DROP TABLE long_insert' | ${CLICKHOUSE_CURL} -sSg "${CLICKHOUSE_URL}" -d @- diff --git a/tests/queries/0_stateless/00062_replicated_merge_tree_alter_zookeeper.reference b/tests/queries/0_stateless/00062_replicated_merge_tree_alter_zookeeper.reference index 69887db71c1..cb61ab3e9b9 100644 --- a/tests/queries/0_stateless/00062_replicated_merge_tree_alter_zookeeper.reference +++ b/tests/queries/0_stateless/00062_replicated_merge_tree_alter_zookeeper.reference @@ -1,22 +1,22 @@ d Date k UInt64 i32 Int32 -CREATE TABLE test.replicated_alter1\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r1\', d, k, 8192) +CREATE TABLE default.replicated_alter1\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r1\', d, k, 8192) d Date k UInt64 i32 Int32 -CREATE TABLE test.replicated_alter2\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r2\', d, k, 8192) +CREATE TABLE default.replicated_alter2\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r2\', d, k, 8192) 2015-01-01 10 42 d Date k UInt64 i32 Int32 dt DateTime(\'UTC\') -CREATE TABLE test.replicated_alter1\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\')\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r1\', d, k, 8192) +CREATE TABLE default.replicated_alter1\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\')\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r1\', d, k, 8192) d Date k UInt64 i32 Int32 dt DateTime(\'UTC\') -CREATE TABLE test.replicated_alter2\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\')\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r2\', d, k, 8192) +CREATE TABLE default.replicated_alter2\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\')\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r2\', d, k, 8192) 2015-01-01 9 41 1992-01-01 08:00:00 2015-01-01 10 42 1970-01-01 00:00:00 d Date @@ -25,14 +25,14 @@ i32 Int32 dt DateTime(\'UTC\') n.ui8 Array(UInt8) n.s Array(String) -CREATE TABLE test.replicated_alter1\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\'),\n `n.ui8` Array(UInt8),\n `n.s` Array(String)\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r1\', d, k, 8192) +CREATE TABLE default.replicated_alter1\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\'),\n `n.ui8` Array(UInt8),\n `n.s` Array(String)\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r1\', d, k, 8192) d Date k UInt64 i32 Int32 dt DateTime(\'UTC\') n.ui8 Array(UInt8) n.s Array(String) -CREATE TABLE test.replicated_alter2\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\'),\n `n.ui8` Array(UInt8),\n `n.s` Array(String)\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r2\', d, k, 8192) +CREATE TABLE default.replicated_alter2\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\'),\n `n.ui8` Array(UInt8),\n `n.s` Array(String)\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r2\', d, k, 8192) 2015-01-01 8 40 2012-12-12 12:12:12 [1,2,3] ['12','13','14'] 2015-01-01 9 41 1992-01-01 08:00:00 [] [] 2015-01-01 10 42 1970-01-01 00:00:00 [] [] @@ -43,7 +43,7 @@ dt DateTime(\'UTC\') n.ui8 Array(UInt8) n.s Array(String) n.d Array(Date) -CREATE TABLE test.replicated_alter1\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\'),\n `n.ui8` Array(UInt8),\n `n.s` Array(String),\n `n.d` Array(Date)\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r1\', d, k, 8192) +CREATE TABLE default.replicated_alter1\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\'),\n `n.ui8` Array(UInt8),\n `n.s` Array(String),\n `n.d` Array(Date)\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r1\', d, k, 8192) d Date k UInt64 i32 Int32 @@ -51,7 +51,7 @@ dt DateTime(\'UTC\') n.ui8 Array(UInt8) n.s Array(String) n.d Array(Date) -CREATE TABLE test.replicated_alter2\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\'),\n `n.ui8` Array(UInt8),\n `n.s` Array(String),\n `n.d` Array(Date)\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r2\', d, k, 8192) +CREATE TABLE default.replicated_alter2\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\'),\n `n.ui8` Array(UInt8),\n `n.s` Array(String),\n `n.d` Array(Date)\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r2\', d, k, 8192) 2015-01-01 7 39 2014-07-14 13:26:50 [10,20,30] ['120','130','140'] ['2000-01-01','2000-01-01','2000-01-03'] 2015-01-01 8 40 2012-12-12 12:12:12 [1,2,3] ['12','13','14'] ['1970-01-01','1970-01-01','1970-01-01'] 2015-01-01 9 41 1992-01-01 08:00:00 [] [] [] @@ -64,7 +64,7 @@ n.ui8 Array(UInt8) n.s Array(String) n.d Array(Date) s String DEFAULT \'0\' -CREATE TABLE test.replicated_alter1\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\'),\n `n.ui8` Array(UInt8),\n `n.s` Array(String),\n `n.d` Array(Date),\n `s` String DEFAULT \'0\'\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r1\', d, k, 8192) +CREATE TABLE default.replicated_alter1\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\'),\n `n.ui8` Array(UInt8),\n `n.s` Array(String),\n `n.d` Array(Date),\n `s` String DEFAULT \'0\'\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r1\', d, k, 8192) d Date k UInt64 i32 Int32 @@ -73,7 +73,7 @@ n.ui8 Array(UInt8) n.s Array(String) n.d Array(Date) s String DEFAULT \'0\' -CREATE TABLE test.replicated_alter2\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\'),\n `n.ui8` Array(UInt8),\n `n.s` Array(String),\n `n.d` Array(Date),\n `s` String DEFAULT \'0\'\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r2\', d, k, 8192) +CREATE TABLE default.replicated_alter2\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\'),\n `n.ui8` Array(UInt8),\n `n.s` Array(String),\n `n.d` Array(Date),\n `s` String DEFAULT \'0\'\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r2\', d, k, 8192) 2015-01-01 6 38 2014-07-15 13:26:50 [10,20,30] ['asd','qwe','qwe'] ['2000-01-01','2000-01-01','2000-01-03'] 100500 2015-01-01 7 39 2014-07-14 13:26:50 [10,20,30] ['120','130','140'] ['2000-01-01','2000-01-01','2000-01-03'] 0 2015-01-01 8 40 2012-12-12 12:12:12 [1,2,3] ['12','13','14'] ['1970-01-01','1970-01-01','1970-01-01'] 0 @@ -86,7 +86,7 @@ dt DateTime(\'UTC\') n.ui8 Array(UInt8) n.s Array(String) s Int64 DEFAULT \'0\' -CREATE TABLE test.replicated_alter1\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\'),\n `n.ui8` Array(UInt8),\n `n.s` Array(String),\n `s` Int64 DEFAULT \'0\'\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r1\', d, k, 8192) +CREATE TABLE default.replicated_alter1\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\'),\n `n.ui8` Array(UInt8),\n `n.s` Array(String),\n `s` Int64 DEFAULT \'0\'\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r1\', d, k, 8192) d Date k UInt64 i32 Int32 @@ -94,7 +94,7 @@ dt DateTime(\'UTC\') n.ui8 Array(UInt8) n.s Array(String) s Int64 DEFAULT \'0\' -CREATE TABLE test.replicated_alter2\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\'),\n `n.ui8` Array(UInt8),\n `n.s` Array(String),\n `s` Int64 DEFAULT \'0\'\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r2\', d, k, 8192) +CREATE TABLE default.replicated_alter2\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\'),\n `n.ui8` Array(UInt8),\n `n.s` Array(String),\n `s` Int64 DEFAULT \'0\'\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r2\', d, k, 8192) 2015-01-01 6 38 2014-07-15 13:26:50 [10,20,30] ['asd','qwe','qwe'] 100500 2015-01-01 7 39 2014-07-14 13:26:50 [10,20,30] ['120','130','140'] 0 2015-01-01 8 40 2012-12-12 12:12:12 [1,2,3] ['12','13','14'] 0 @@ -108,7 +108,7 @@ n.ui8 Array(UInt8) n.s Array(String) s UInt32 DEFAULT \'0\' n.d Array(Date) -CREATE TABLE test.replicated_alter1\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\'),\n `n.ui8` Array(UInt8),\n `n.s` Array(String),\n `s` UInt32 DEFAULT \'0\',\n `n.d` Array(Date)\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r1\', d, k, 8192) +CREATE TABLE default.replicated_alter1\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\'),\n `n.ui8` Array(UInt8),\n `n.s` Array(String),\n `s` UInt32 DEFAULT \'0\',\n `n.d` Array(Date)\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r1\', d, k, 8192) d Date k UInt64 i32 Int32 @@ -117,7 +117,7 @@ n.ui8 Array(UInt8) n.s Array(String) s UInt32 DEFAULT \'0\' n.d Array(Date) -CREATE TABLE test.replicated_alter2\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\'),\n `n.ui8` Array(UInt8),\n `n.s` Array(String),\n `s` UInt32 DEFAULT \'0\',\n `n.d` Array(Date)\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r2\', d, k, 8192) +CREATE TABLE default.replicated_alter2\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\'),\n `n.ui8` Array(UInt8),\n `n.s` Array(String),\n `s` UInt32 DEFAULT \'0\',\n `n.d` Array(Date)\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r2\', d, k, 8192) 2015-01-01 6 38 2014-07-15 13:26:50 [10,20,30] ['asd','qwe','qwe'] 100500 ['1970-01-01','1970-01-01','1970-01-01'] 2015-01-01 7 39 2014-07-14 13:26:50 [10,20,30] ['120','130','140'] 0 ['1970-01-01','1970-01-01','1970-01-01'] 2015-01-01 8 40 2012-12-12 12:12:12 [1,2,3] ['12','13','14'] 0 ['1970-01-01','1970-01-01','1970-01-01'] @@ -129,14 +129,14 @@ i32 Int32 dt DateTime(\'UTC\') n.s Array(String) s UInt32 DEFAULT \'0\' -CREATE TABLE test.replicated_alter1\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\'),\n `n.s` Array(String),\n `s` UInt32 DEFAULT \'0\'\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r1\', d, k, 8192) +CREATE TABLE default.replicated_alter1\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\'),\n `n.s` Array(String),\n `s` UInt32 DEFAULT \'0\'\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r1\', d, k, 8192) d Date k UInt64 i32 Int32 dt DateTime(\'UTC\') n.s Array(String) s UInt32 DEFAULT \'0\' -CREATE TABLE test.replicated_alter2\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\'),\n `n.s` Array(String),\n `s` UInt32 DEFAULT \'0\'\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r2\', d, k, 8192) +CREATE TABLE default.replicated_alter2\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\'),\n `n.s` Array(String),\n `s` UInt32 DEFAULT \'0\'\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r2\', d, k, 8192) 2015-01-01 6 38 2014-07-15 13:26:50 ['asd','qwe','qwe'] 100500 2015-01-01 7 39 2014-07-14 13:26:50 ['120','130','140'] 0 2015-01-01 8 40 2012-12-12 12:12:12 ['12','13','14'] 0 @@ -147,13 +147,13 @@ k UInt64 i32 Int32 dt DateTime(\'UTC\') s UInt32 DEFAULT \'0\' -CREATE TABLE test.replicated_alter1\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\'),\n `s` UInt32 DEFAULT \'0\'\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r1\', d, k, 8192) +CREATE TABLE default.replicated_alter1\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\'),\n `s` UInt32 DEFAULT \'0\'\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r1\', d, k, 8192) d Date k UInt64 i32 Int32 dt DateTime(\'UTC\') s UInt32 DEFAULT \'0\' -CREATE TABLE test.replicated_alter2\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\'),\n `s` UInt32 DEFAULT \'0\'\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r2\', d, k, 8192) +CREATE TABLE default.replicated_alter2\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\'),\n `s` UInt32 DEFAULT \'0\'\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r2\', d, k, 8192) 2015-01-01 6 38 2014-07-15 13:26:50 100500 2015-01-01 7 39 2014-07-14 13:26:50 0 2015-01-01 8 40 2012-12-12 12:12:12 0 @@ -166,7 +166,7 @@ dt DateTime(\'UTC\') s UInt32 DEFAULT \'0\' n.s Array(String) n.d Array(Date) -CREATE TABLE test.replicated_alter1\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\'),\n `s` UInt32 DEFAULT \'0\',\n `n.s` Array(String),\n `n.d` Array(Date)\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r1\', d, k, 8192) +CREATE TABLE default.replicated_alter1\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\'),\n `s` UInt32 DEFAULT \'0\',\n `n.s` Array(String),\n `n.d` Array(Date)\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r1\', d, k, 8192) d Date k UInt64 i32 Int32 @@ -174,7 +174,7 @@ dt DateTime(\'UTC\') s UInt32 DEFAULT \'0\' n.s Array(String) n.d Array(Date) -CREATE TABLE test.replicated_alter2\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\'),\n `s` UInt32 DEFAULT \'0\',\n `n.s` Array(String),\n `n.d` Array(Date)\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r2\', d, k, 8192) +CREATE TABLE default.replicated_alter2\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\'),\n `s` UInt32 DEFAULT \'0\',\n `n.s` Array(String),\n `n.d` Array(Date)\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r2\', d, k, 8192) 2015-01-01 6 38 2014-07-15 13:26:50 100500 [] [] 2015-01-01 7 39 2014-07-14 13:26:50 0 [] [] 2015-01-01 8 40 2012-12-12 12:12:12 0 [] [] @@ -185,13 +185,13 @@ k UInt64 i32 Int32 dt DateTime(\'UTC\') s UInt32 DEFAULT \'0\' -CREATE TABLE test.replicated_alter1\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\'),\n `s` UInt32 DEFAULT \'0\'\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r1\', d, k, 8192) +CREATE TABLE default.replicated_alter1\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\'),\n `s` UInt32 DEFAULT \'0\'\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r1\', d, k, 8192) d Date k UInt64 i32 Int32 dt DateTime(\'UTC\') s UInt32 DEFAULT \'0\' -CREATE TABLE test.replicated_alter2\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\'),\n `s` UInt32 DEFAULT \'0\'\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r2\', d, k, 8192) +CREATE TABLE default.replicated_alter2\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` DateTime(\'UTC\'),\n `s` UInt32 DEFAULT \'0\'\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r2\', d, k, 8192) 2015-01-01 6 38 2014-07-15 13:26:50 100500 2015-01-01 7 39 2014-07-14 13:26:50 0 2015-01-01 8 40 2012-12-12 12:12:12 0 @@ -202,13 +202,13 @@ k UInt64 i32 Int32 dt Date s DateTime(\'UTC\') DEFAULT \'1970-01-01 00:00:00\' -CREATE TABLE test.replicated_alter1\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` Date,\n `s` DateTime(\'UTC\') DEFAULT \'1970-01-01 00:00:00\'\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r1\', d, k, 8192) +CREATE TABLE default.replicated_alter1\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` Date,\n `s` DateTime(\'UTC\') DEFAULT \'1970-01-01 00:00:00\'\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r1\', d, k, 8192) d Date k UInt64 i32 Int32 dt Date s DateTime(\'UTC\') DEFAULT \'1970-01-01 00:00:00\' -CREATE TABLE test.replicated_alter2\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` Date,\n `s` DateTime(\'UTC\') DEFAULT \'1970-01-01 00:00:00\'\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r2\', d, k, 8192) +CREATE TABLE default.replicated_alter2\n(\n `d` Date,\n `k` UInt64,\n `i32` Int32,\n `dt` Date,\n `s` DateTime(\'UTC\') DEFAULT \'1970-01-01 00:00:00\'\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00062/alter\', \'r2\', d, k, 8192) 2015-01-01 6 38 2014-07-15 1970-01-02 03:55:00 2015-01-01 7 39 2014-07-14 1970-01-01 00:00:00 2015-01-01 8 40 2012-12-12 1970-01-01 00:00:00 diff --git a/tests/queries/0_stateless/00062_replicated_merge_tree_alter_zookeeper.sql b/tests/queries/0_stateless/00062_replicated_merge_tree_alter_zookeeper.sql index 2a933d21eeb..ac56b3416cd 100644 --- a/tests/queries/0_stateless/00062_replicated_merge_tree_alter_zookeeper.sql +++ b/tests/queries/0_stateless/00062_replicated_merge_tree_alter_zookeeper.sql @@ -1,110 +1,110 @@ -DROP TABLE IF EXISTS test.replicated_alter1; -DROP TABLE IF EXISTS test.replicated_alter2; +DROP TABLE IF EXISTS replicated_alter1; +DROP TABLE IF EXISTS replicated_alter2; SET replication_alter_partitions_sync = 2; -CREATE TABLE test.replicated_alter1 (d Date, k UInt64, i32 Int32) ENGINE=ReplicatedMergeTree('/clickhouse/tables/test_00062/alter', 'r1', d, k, 8192); -CREATE TABLE test.replicated_alter2 (d Date, k UInt64, i32 Int32) ENGINE=ReplicatedMergeTree('/clickhouse/tables/test_00062/alter', 'r2', d, k, 8192); +CREATE TABLE replicated_alter1 (d Date, k UInt64, i32 Int32) ENGINE=ReplicatedMergeTree('/clickhouse/tables/test_00062/alter', 'r1', d, k, 8192); +CREATE TABLE replicated_alter2 (d Date, k UInt64, i32 Int32) ENGINE=ReplicatedMergeTree('/clickhouse/tables/test_00062/alter', 'r2', d, k, 8192); -INSERT INTO test.replicated_alter1 VALUES ('2015-01-01', 10, 42); +INSERT INTO replicated_alter1 VALUES ('2015-01-01', 10, 42); -DESC TABLE test.replicated_alter1; -SHOW CREATE TABLE test.replicated_alter1; -DESC TABLE test.replicated_alter2; -SHOW CREATE TABLE test.replicated_alter2; -SELECT * FROM test.replicated_alter1 ORDER BY k; +DESC TABLE replicated_alter1; +SHOW CREATE TABLE replicated_alter1; +DESC TABLE replicated_alter2; +SHOW CREATE TABLE replicated_alter2; +SELECT * FROM replicated_alter1 ORDER BY k; -ALTER TABLE test.replicated_alter1 ADD COLUMN dt DateTime('UTC'); -INSERT INTO test.replicated_alter1 VALUES ('2015-01-01', 9, 41, '1992-01-01 08:00:00'); +ALTER TABLE replicated_alter1 ADD COLUMN dt DateTime('UTC'); +INSERT INTO replicated_alter1 VALUES ('2015-01-01', 9, 41, '1992-01-01 08:00:00'); -DESC TABLE test.replicated_alter1; -SHOW CREATE TABLE test.replicated_alter1; -DESC TABLE test.replicated_alter2; -SHOW CREATE TABLE test.replicated_alter2; -SELECT * FROM test.replicated_alter1 ORDER BY k; +DESC TABLE replicated_alter1; +SHOW CREATE TABLE replicated_alter1; +DESC TABLE replicated_alter2; +SHOW CREATE TABLE replicated_alter2; +SELECT * FROM replicated_alter1 ORDER BY k; -ALTER TABLE test.replicated_alter1 ADD COLUMN n Nested(ui8 UInt8, s String); -INSERT INTO test.replicated_alter1 VALUES ('2015-01-01', 8, 40, '2012-12-12 12:12:12', [1,2,3], ['12','13','14']); +ALTER TABLE replicated_alter1 ADD COLUMN n Nested(ui8 UInt8, s String); +INSERT INTO replicated_alter1 VALUES ('2015-01-01', 8, 40, '2012-12-12 12:12:12', [1,2,3], ['12','13','14']); -DESC TABLE test.replicated_alter1; -SHOW CREATE TABLE test.replicated_alter1; -DESC TABLE test.replicated_alter2; -SHOW CREATE TABLE test.replicated_alter2; -SELECT * FROM test.replicated_alter1 ORDER BY k; +DESC TABLE replicated_alter1; +SHOW CREATE TABLE replicated_alter1; +DESC TABLE replicated_alter2; +SHOW CREATE TABLE replicated_alter2; +SELECT * FROM replicated_alter1 ORDER BY k; -ALTER TABLE test.replicated_alter1 ADD COLUMN `n.d` Array(Date); -INSERT INTO test.replicated_alter1 VALUES ('2015-01-01', 7, 39, '2014-07-14 13:26:50', [10,20,30], ['120','130','140'],['2000-01-01','2000-01-01','2000-01-03']); +ALTER TABLE replicated_alter1 ADD COLUMN `n.d` Array(Date); +INSERT INTO replicated_alter1 VALUES ('2015-01-01', 7, 39, '2014-07-14 13:26:50', [10,20,30], ['120','130','140'],['2000-01-01','2000-01-01','2000-01-03']); -DESC TABLE test.replicated_alter1; -SHOW CREATE TABLE test.replicated_alter1; -DESC TABLE test.replicated_alter2; -SHOW CREATE TABLE test.replicated_alter2; -SELECT * FROM test.replicated_alter1 ORDER BY k; +DESC TABLE replicated_alter1; +SHOW CREATE TABLE replicated_alter1; +DESC TABLE replicated_alter2; +SHOW CREATE TABLE replicated_alter2; +SELECT * FROM replicated_alter1 ORDER BY k; -ALTER TABLE test.replicated_alter1 ADD COLUMN s String DEFAULT '0'; -INSERT INTO test.replicated_alter1 VALUES ('2015-01-01', 6,38,'2014-07-15 13:26:50',[10,20,30],['asd','qwe','qwe'],['2000-01-01','2000-01-01','2000-01-03'],'100500'); +ALTER TABLE replicated_alter1 ADD COLUMN s String DEFAULT '0'; +INSERT INTO replicated_alter1 VALUES ('2015-01-01', 6,38,'2014-07-15 13:26:50',[10,20,30],['asd','qwe','qwe'],['2000-01-01','2000-01-01','2000-01-03'],'100500'); -DESC TABLE test.replicated_alter1; -SHOW CREATE TABLE test.replicated_alter1; -DESC TABLE test.replicated_alter2; -SHOW CREATE TABLE test.replicated_alter2; -SELECT * FROM test.replicated_alter1 ORDER BY k; +DESC TABLE replicated_alter1; +SHOW CREATE TABLE replicated_alter1; +DESC TABLE replicated_alter2; +SHOW CREATE TABLE replicated_alter2; +SELECT * FROM replicated_alter1 ORDER BY k; -ALTER TABLE test.replicated_alter1 DROP COLUMN `n.d`, MODIFY COLUMN s Int64; +ALTER TABLE replicated_alter1 DROP COLUMN `n.d`, MODIFY COLUMN s Int64; -DESC TABLE test.replicated_alter1; -SHOW CREATE TABLE test.replicated_alter1; -DESC TABLE test.replicated_alter2; -SHOW CREATE TABLE test.replicated_alter2; -SELECT * FROM test.replicated_alter1 ORDER BY k; +DESC TABLE replicated_alter1; +SHOW CREATE TABLE replicated_alter1; +DESC TABLE replicated_alter2; +SHOW CREATE TABLE replicated_alter2; +SELECT * FROM replicated_alter1 ORDER BY k; -ALTER TABLE test.replicated_alter1 ADD COLUMN `n.d` Array(Date), MODIFY COLUMN s UInt32; +ALTER TABLE replicated_alter1 ADD COLUMN `n.d` Array(Date), MODIFY COLUMN s UInt32; -DESC TABLE test.replicated_alter1; -SHOW CREATE TABLE test.replicated_alter1; -DESC TABLE test.replicated_alter2; -SHOW CREATE TABLE test.replicated_alter2; -SELECT * FROM test.replicated_alter1 ORDER BY k; +DESC TABLE replicated_alter1; +SHOW CREATE TABLE replicated_alter1; +DESC TABLE replicated_alter2; +SHOW CREATE TABLE replicated_alter2; +SELECT * FROM replicated_alter1 ORDER BY k; -ALTER TABLE test.replicated_alter1 DROP COLUMN n.ui8, DROP COLUMN n.d; +ALTER TABLE replicated_alter1 DROP COLUMN n.ui8, DROP COLUMN n.d; -DESC TABLE test.replicated_alter1; -SHOW CREATE TABLE test.replicated_alter1; -DESC TABLE test.replicated_alter2; -SHOW CREATE TABLE test.replicated_alter2; -SELECT * FROM test.replicated_alter1 ORDER BY k; +DESC TABLE replicated_alter1; +SHOW CREATE TABLE replicated_alter1; +DESC TABLE replicated_alter2; +SHOW CREATE TABLE replicated_alter2; +SELECT * FROM replicated_alter1 ORDER BY k; -ALTER TABLE test.replicated_alter1 DROP COLUMN n.s; +ALTER TABLE replicated_alter1 DROP COLUMN n.s; -DESC TABLE test.replicated_alter1; -SHOW CREATE TABLE test.replicated_alter1; -DESC TABLE test.replicated_alter2; -SHOW CREATE TABLE test.replicated_alter2; -SELECT * FROM test.replicated_alter1 ORDER BY k; +DESC TABLE replicated_alter1; +SHOW CREATE TABLE replicated_alter1; +DESC TABLE replicated_alter2; +SHOW CREATE TABLE replicated_alter2; +SELECT * FROM replicated_alter1 ORDER BY k; -ALTER TABLE test.replicated_alter1 ADD COLUMN n.s Array(String), ADD COLUMN n.d Array(Date); +ALTER TABLE replicated_alter1 ADD COLUMN n.s Array(String), ADD COLUMN n.d Array(Date); -DESC TABLE test.replicated_alter1; -SHOW CREATE TABLE test.replicated_alter1; -DESC TABLE test.replicated_alter2; -SHOW CREATE TABLE test.replicated_alter2; -SELECT * FROM test.replicated_alter1 ORDER BY k; +DESC TABLE replicated_alter1; +SHOW CREATE TABLE replicated_alter1; +DESC TABLE replicated_alter2; +SHOW CREATE TABLE replicated_alter2; +SELECT * FROM replicated_alter1 ORDER BY k; -ALTER TABLE test.replicated_alter1 DROP COLUMN n; +ALTER TABLE replicated_alter1 DROP COLUMN n; -DESC TABLE test.replicated_alter1; -SHOW CREATE TABLE test.replicated_alter1; -DESC TABLE test.replicated_alter2; -SHOW CREATE TABLE test.replicated_alter2; -SELECT * FROM test.replicated_alter1 ORDER BY k; +DESC TABLE replicated_alter1; +SHOW CREATE TABLE replicated_alter1; +DESC TABLE replicated_alter2; +SHOW CREATE TABLE replicated_alter2; +SELECT * FROM replicated_alter1 ORDER BY k; -ALTER TABLE test.replicated_alter1 MODIFY COLUMN dt Date, MODIFY COLUMN s DateTime('UTC') DEFAULT '1970-01-01 00:00:00'; +ALTER TABLE replicated_alter1 MODIFY COLUMN dt Date, MODIFY COLUMN s DateTime('UTC') DEFAULT '1970-01-01 00:00:00'; -DESC TABLE test.replicated_alter1; -SHOW CREATE TABLE test.replicated_alter1; -DESC TABLE test.replicated_alter2; -SHOW CREATE TABLE test.replicated_alter2; -SELECT * FROM test.replicated_alter1 ORDER BY k; +DESC TABLE replicated_alter1; +SHOW CREATE TABLE replicated_alter1; +DESC TABLE replicated_alter2; +SHOW CREATE TABLE replicated_alter2; +SELECT * FROM replicated_alter1 ORDER BY k; -DROP TABLE test.replicated_alter1; -DROP TABLE test.replicated_alter2; +DROP TABLE replicated_alter1; +DROP TABLE replicated_alter2; diff --git a/tests/queries/0_stateless/00070_insert_fewer_columns_http.sh b/tests/queries/0_stateless/00070_insert_fewer_columns_http.sh index 853d4e161c9..0cf5f95d3d9 100755 --- a/tests/queries/0_stateless/00070_insert_fewer_columns_http.sh +++ b/tests/queries/0_stateless/00070_insert_fewer_columns_http.sh @@ -8,3 +8,4 @@ echo 'DROP TABLE IF EXISTS insert_fewer_columns' | ${ echo 'CREATE TABLE insert_fewer_columns (a UInt8, b UInt8) ENGINE = Memory' | ${CLICKHOUSE_CURL} -sSg "${CLICKHOUSE_URL}" -d @- echo 'INSERT INTO insert_fewer_columns (a) VALUES (1), (2)' | ${CLICKHOUSE_CURL} -sSg "${CLICKHOUSE_URL}" -d @- echo 'SELECT * FROM insert_fewer_columns' | ${CLICKHOUSE_CURL} -sSg "${CLICKHOUSE_URL}" -d @- +echo 'DROP TABLE insert_fewer_columns' | ${CLICKHOUSE_CURL} -sSg "${CLICKHOUSE_URL}" -d @- diff --git a/tests/queries/0_stateless/00302_http_compression.reference b/tests/queries/0_stateless/00302_http_compression.reference index b2ab0bdb76c..f3d28359515 100644 --- a/tests/queries/0_stateless/00302_http_compression.reference +++ b/tests/queries/0_stateless/00302_http_compression.reference @@ -58,13 +58,26 @@ 7 8 9 +0 +1 +2 +3 +4 +5 +6 +7 +8 +9 < Content-Encoding: gzip < Content-Encoding: deflate < Content-Encoding: gzip < Content-Encoding: br +< Content-Encoding: xz 1 1 1 +1 +Hello, world Hello, world Hello, world Hello, world diff --git a/tests/queries/0_stateless/00302_http_compression.sh b/tests/queries/0_stateless/00302_http_compression.sh index f3e10028dfb..846a23a54c9 100755 --- a/tests/queries/0_stateless/00302_http_compression.sh +++ b/tests/queries/0_stateless/00302_http_compression.sh @@ -9,6 +9,7 @@ ${CLICKHOUSE_CURL} -sS "${CLICKHOUSE_URL}&enable_http_compression=1" -H 'Accept- ${CLICKHOUSE_CURL} -sS "${CLICKHOUSE_URL}&enable_http_compression=1" -H 'Accept-Encoding: gzip, deflate' -d 'SELECT number FROM system.numbers LIMIT 10' | gzip -d; ${CLICKHOUSE_CURL} -sS "${CLICKHOUSE_URL}&enable_http_compression=1" -H 'Accept-Encoding: zip, eflate' -d 'SELECT number FROM system.numbers LIMIT 10'; ${CLICKHOUSE_CURL} -sS "${CLICKHOUSE_URL}&enable_http_compression=1" -H 'Accept-Encoding: br' -d 'SELECT number FROM system.numbers LIMIT 10' | brotli -d; +${CLICKHOUSE_CURL} -sS "${CLICKHOUSE_URL}&enable_http_compression=1" -H 'Accept-Encoding: xz' -d 'SELECT number FROM system.numbers LIMIT 10' | xz -d; ${CLICKHOUSE_CURL} -vsS "${CLICKHOUSE_URL}&enable_http_compression=1" -d 'SELECT number FROM system.numbers LIMIT 10' 2>&1 | grep --text '< Content-Encoding'; ${CLICKHOUSE_CURL} -vsS "${CLICKHOUSE_URL}&enable_http_compression=1" -H 'Accept-Encoding: gzip' -d 'SELECT number FROM system.numbers LIMIT 10' 2>&1 | grep --text '< Content-Encoding'; @@ -16,14 +17,17 @@ ${CLICKHOUSE_CURL} -vsS "${CLICKHOUSE_URL}&enable_http_compression=1" -H 'Accept ${CLICKHOUSE_CURL} -vsS "${CLICKHOUSE_URL}&enable_http_compression=1" -H 'Accept-Encoding: gzip, deflate' -d 'SELECT number FROM system.numbers LIMIT 10' 2>&1 | grep --text '< Content-Encoding'; ${CLICKHOUSE_CURL} -vsS "${CLICKHOUSE_URL}&enable_http_compression=1" -H 'Accept-Encoding: zip, eflate' -d 'SELECT number FROM system.numbers LIMIT 10' 2>&1 | grep --text '< Content-Encoding'; ${CLICKHOUSE_CURL} -vsS "${CLICKHOUSE_URL}&enable_http_compression=1" -H 'Accept-Encoding: br' -d 'SELECT number FROM system.numbers LIMIT 10' 2>&1 | grep --text '< Content-Encoding'; +${CLICKHOUSE_CURL} -vsS "${CLICKHOUSE_URL}&enable_http_compression=1" -H 'Accept-Encoding: xz' -d 'SELECT number FROM system.numbers LIMIT 10' 2>&1 | grep --text '< Content-Encoding'; echo "SELECT 1" | ${CLICKHOUSE_CURL} -sS --data-binary @- "${CLICKHOUSE_URL}"; echo "SELECT 1" | gzip -c | ${CLICKHOUSE_CURL} -sS --data-binary @- -H 'Content-Encoding: gzip' "${CLICKHOUSE_URL}"; echo "SELECT 1" | brotli | ${CLICKHOUSE_CURL} -sS --data-binary @- -H 'Content-Encoding: br' "${CLICKHOUSE_URL}"; +echo "SELECT 1" | xz -c | ${CLICKHOUSE_CURL} -sS --data-binary @- -H 'Content-Encoding: xz' "${CLICKHOUSE_URL}"; echo "'Hello, world'" | ${CLICKHOUSE_CURL} -sS --data-binary @- "${CLICKHOUSE_URL}&query=SELECT"; echo "'Hello, world'" | gzip -c | ${CLICKHOUSE_CURL} -sS --data-binary @- -H 'Content-Encoding: gzip' "${CLICKHOUSE_URL}&query=SELECT"; echo "'Hello, world'" | brotli | ${CLICKHOUSE_CURL} -sS --data-binary @- -H 'Content-Encoding: br' "${CLICKHOUSE_URL}&query=SELECT"; +echo "'Hello, world'" | xz -c | ${CLICKHOUSE_CURL} -sS --data-binary @- -H 'Content-Encoding: xz' "${CLICKHOUSE_URL}&query=SELECT"; ${CLICKHOUSE_CURL} -sS "${CLICKHOUSE_URL}&enable_http_compression=1" -H 'Accept-Encoding: gzip' -d 'SELECT number FROM system.numbers LIMIT 0' | wc -c; diff --git a/tests/queries/0_stateless/00386_long_in_pk.python b/tests/queries/0_stateless/00386_long_in_pk.python index ab5fc50d8e3..e33bb254c60 100644 --- a/tests/queries/0_stateless/00386_long_in_pk.python +++ b/tests/queries/0_stateless/00386_long_in_pk.python @@ -50,6 +50,8 @@ def main(): print(resp.text) break + requests.post(url, data='drop table tab_00386') + if __name__ == "__main__": main() diff --git a/tests/queries/0_stateless/00446_clear_column_in_partition_zookeeper.sql b/tests/queries/0_stateless/00446_clear_column_in_partition_zookeeper.sql index bd6c12ffce4..5d8c4de1c06 100644 --- a/tests/queries/0_stateless/00446_clear_column_in_partition_zookeeper.sql +++ b/tests/queries/0_stateless/00446_clear_column_in_partition_zookeeper.sql @@ -71,3 +71,6 @@ OPTIMIZE TABLE clear_column1 PARTITION '200002'; ALTER TABLE clear_column1 CLEAR COLUMN s IN PARTITION '200012', CLEAR COLUMN i IN PARTITION '200012'; -- Drop empty partition also Ok ALTER TABLE clear_column1 DROP PARTITION '200012', DROP PARTITION '200011'; + +DROP TABLE clear_column1; +DROP TABLE clear_column2; diff --git a/tests/queries/0_stateless/00509_extended_storage_definition_syntax_zookeeper.sql b/tests/queries/0_stateless/00509_extended_storage_definition_syntax_zookeeper.sql index 4a3dd2981cd..4ed053f5953 100644 --- a/tests/queries/0_stateless/00509_extended_storage_definition_syntax_zookeeper.sql +++ b/tests/queries/0_stateless/00509_extended_storage_definition_syntax_zookeeper.sql @@ -1,81 +1,81 @@ SELECT '*** Replicated with sampling ***'; -DROP TABLE IF EXISTS test.replicated_with_sampling; +DROP TABLE IF EXISTS replicated_with_sampling; -CREATE TABLE test.replicated_with_sampling(x UInt8) +CREATE TABLE replicated_with_sampling(x UInt8) ENGINE ReplicatedMergeTree('/clickhouse/tables/test_00509/replicated_with_sampling', 'r1') ORDER BY x SAMPLE BY x; -INSERT INTO test.replicated_with_sampling VALUES (1), (128); -SELECT sum(x) FROM test.replicated_with_sampling SAMPLE 1/2; +INSERT INTO replicated_with_sampling VALUES (1), (128); +SELECT sum(x) FROM replicated_with_sampling SAMPLE 1/2; -DROP TABLE test.replicated_with_sampling; +DROP TABLE replicated_with_sampling; SELECT '*** Replacing with implicit version ***'; -DROP TABLE IF EXISTS test.replacing; +DROP TABLE IF EXISTS replacing; -CREATE TABLE test.replacing(d Date, x UInt32, s String) ENGINE = ReplacingMergeTree ORDER BY x PARTITION BY d; +CREATE TABLE replacing(d Date, x UInt32, s String) ENGINE = ReplacingMergeTree ORDER BY x PARTITION BY d; -INSERT INTO test.replacing VALUES ('2017-10-23', 1, 'a'); -INSERT INTO test.replacing VALUES ('2017-10-23', 1, 'b'); -INSERT INTO test.replacing VALUES ('2017-10-23', 1, 'c'); +INSERT INTO replacing VALUES ('2017-10-23', 1, 'a'); +INSERT INTO replacing VALUES ('2017-10-23', 1, 'b'); +INSERT INTO replacing VALUES ('2017-10-23', 1, 'c'); -OPTIMIZE TABLE test.replacing PARTITION '2017-10-23' FINAL; +OPTIMIZE TABLE replacing PARTITION '2017-10-23' FINAL; -SELECT * FROM test.replacing; +SELECT * FROM replacing; -DROP TABLE test.replacing; +DROP TABLE replacing; SELECT '*** Replicated Collapsing ***'; -DROP TABLE IF EXISTS test.replicated_collapsing; +DROP TABLE IF EXISTS replicated_collapsing; -CREATE TABLE test.replicated_collapsing(d Date, x UInt32, sign Int8) +CREATE TABLE replicated_collapsing(d Date, x UInt32, sign Int8) ENGINE = ReplicatedCollapsingMergeTree('/clickhouse/tables/test_00509/replicated_collapsing', 'r1', sign) PARTITION BY toYYYYMM(d) ORDER BY d; -INSERT INTO test.replicated_collapsing VALUES ('2017-10-23', 1, 1); -INSERT INTO test.replicated_collapsing VALUES ('2017-10-23', 1, -1), ('2017-10-23', 2, 1); +INSERT INTO replicated_collapsing VALUES ('2017-10-23', 1, 1); +INSERT INTO replicated_collapsing VALUES ('2017-10-23', 1, -1), ('2017-10-23', 2, 1); -SYSTEM SYNC REPLICA test.replicated_collapsing; -OPTIMIZE TABLE test.replicated_collapsing PARTITION 201710 FINAL; +SYSTEM SYNC REPLICA replicated_collapsing; +OPTIMIZE TABLE replicated_collapsing PARTITION 201710 FINAL; -SELECT * FROM test.replicated_collapsing; +SELECT * FROM replicated_collapsing; -DROP TABLE test.replicated_collapsing; +DROP TABLE replicated_collapsing; SELECT '*** Replicated VersionedCollapsing ***'; -DROP TABLE IF EXISTS test.replicated_versioned_collapsing; +DROP TABLE IF EXISTS replicated_versioned_collapsing; -CREATE TABLE test.replicated_versioned_collapsing(d Date, x UInt32, sign Int8, version UInt8) +CREATE TABLE replicated_versioned_collapsing(d Date, x UInt32, sign Int8, version UInt8) ENGINE = ReplicatedVersionedCollapsingMergeTree('/clickhouse/tables/test_00509/replicated_versioned_collapsing', 'r1', sign, version) PARTITION BY toYYYYMM(d) ORDER BY (d, version); -INSERT INTO test.replicated_versioned_collapsing VALUES ('2017-10-23', 1, 1, 0); -INSERT INTO test.replicated_versioned_collapsing VALUES ('2017-10-23', 1, -1, 0), ('2017-10-23', 2, 1, 0); -INSERT INTO test.replicated_versioned_collapsing VALUES ('2017-10-23', 1, -1, 1), ('2017-10-23', 2, 1, 2); +INSERT INTO replicated_versioned_collapsing VALUES ('2017-10-23', 1, 1, 0); +INSERT INTO replicated_versioned_collapsing VALUES ('2017-10-23', 1, -1, 0), ('2017-10-23', 2, 1, 0); +INSERT INTO replicated_versioned_collapsing VALUES ('2017-10-23', 1, -1, 1), ('2017-10-23', 2, 1, 2); -SYSTEM SYNC REPLICA test.replicated_versioned_collapsing; -OPTIMIZE TABLE test.replicated_versioned_collapsing PARTITION 201710 FINAL; +SYSTEM SYNC REPLICA replicated_versioned_collapsing; +OPTIMIZE TABLE replicated_versioned_collapsing PARTITION 201710 FINAL; -SELECT * FROM test.replicated_versioned_collapsing; +SELECT * FROM replicated_versioned_collapsing; -DROP TABLE test.replicated_versioned_collapsing; +DROP TABLE replicated_versioned_collapsing; SELECT '*** Table definition with SETTINGS ***'; -DROP TABLE IF EXISTS test.with_settings; +DROP TABLE IF EXISTS with_settings; -CREATE TABLE test.with_settings(x UInt32) +CREATE TABLE with_settings(x UInt32) ENGINE ReplicatedMergeTree('/clickhouse/tables/test_00509/with_settings', 'r1') ORDER BY x SETTINGS replicated_can_become_leader = 0; SELECT sleep(1); -- If replicated_can_become_leader were true, this replica would become the leader after 1 second. -SELECT is_leader FROM system.replicas WHERE database = 'test' AND table = 'with_settings'; +SELECT is_leader FROM system.replicas WHERE database = currentDatabase() AND table = 'with_settings'; -DROP TABLE test.with_settings; +DROP TABLE with_settings; diff --git a/tests/queries/0_stateless/00643_cast_zookeeper.reference b/tests/queries/0_stateless/00643_cast_zookeeper.reference index eee4b660bda..9123463de1a 100644 --- a/tests/queries/0_stateless/00643_cast_zookeeper.reference +++ b/tests/queries/0_stateless/00643_cast_zookeeper.reference @@ -1,4 +1,4 @@ -CREATE TABLE test.cast1 +CREATE TABLE default.cast1 ( `x` UInt8, `e` Enum8('hello' = 1, 'world' = 2) DEFAULT CAST(x, 'Enum8(\'hello\' = 1, \'world\' = 2)') diff --git a/tests/queries/0_stateless/00643_cast_zookeeper.sql b/tests/queries/0_stateless/00643_cast_zookeeper.sql index db439425eb8..c52d44bd88b 100644 --- a/tests/queries/0_stateless/00643_cast_zookeeper.sql +++ b/tests/queries/0_stateless/00643_cast_zookeeper.sql @@ -1,7 +1,7 @@ -DROP TABLE IF EXISTS test.cast1; -DROP TABLE IF EXISTS test.cast2; +DROP TABLE IF EXISTS cast1; +DROP TABLE IF EXISTS cast2; -CREATE TABLE test.cast1 +CREATE TABLE cast1 ( x UInt8, e Enum8 @@ -22,17 +22,17 @@ CREATE TABLE test.cast1 ) ) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test_00643/cast', 'r1') ORDER BY e; -SHOW CREATE TABLE test.cast1 FORMAT TSVRaw; -DESC TABLE test.cast1; +SHOW CREATE TABLE cast1 FORMAT TSVRaw; +DESC TABLE cast1; -INSERT INTO test.cast1 (x) VALUES (1); -SELECT * FROM test.cast1; +INSERT INTO cast1 (x) VALUES (1); +SELECT * FROM cast1; -CREATE TABLE test.cast2 AS test.cast1 ENGINE = ReplicatedMergeTree('/clickhouse/tables/test_00643/cast', 'r2') ORDER BY e; +CREATE TABLE cast2 AS cast1 ENGINE = ReplicatedMergeTree('/clickhouse/tables/test_00643/cast', 'r2') ORDER BY e; -SYSTEM SYNC REPLICA test.cast2; +SYSTEM SYNC REPLICA cast2; -SELECT * FROM test.cast2; +SELECT * FROM cast2; -DROP TABLE test.cast1; -DROP TABLE test.cast2; +DROP TABLE cast1; +DROP TABLE cast2; diff --git a/tests/queries/0_stateless/00714_create_temporary_table_with_in_clause.sql b/tests/queries/0_stateless/00714_create_temporary_table_with_in_clause.sql index 751f766d0a1..e56c3bd1d7b 100644 --- a/tests/queries/0_stateless/00714_create_temporary_table_with_in_clause.sql +++ b/tests/queries/0_stateless/00714_create_temporary_table_with_in_clause.sql @@ -1,10 +1,10 @@ DROP TEMPORARY TABLE IF EXISTS temporary_table; -DROP TEMPORARY TABLE IF EXISTS test_merge_1; -DROP TEMPORARY TABLE IF EXISTS test_merge_2; +DROP TABLE IF EXISTS test_merge_1; +DROP TABLE IF EXISTS test_merge_2; CREATE TABLE test_merge_1(id UInt64) ENGINE = Log; CREATE TABLE test_merge_2(id UInt64) ENGINE = Log; CREATE TEMPORARY TABLE temporary_table AS SELECT * FROM numbers(1) WHERE number NOT IN (SELECT id FROM merge(currentDatabase(), 'test_merge_1|test_merge_2')); SELECT * FROM temporary_table; DROP TEMPORARY TABLE IF EXISTS temporary_table; -DROP TEMPORARY TABLE IF EXISTS test_merge_1; -DROP TEMPORARY TABLE IF EXISTS test_merge_2; +DROP TABLE IF EXISTS test_merge_1; +DROP TABLE IF EXISTS test_merge_2; diff --git a/tests/queries/0_stateless/00719_insert_block_without_column.sh b/tests/queries/0_stateless/00719_insert_block_without_column.sh index 5b7619e7b85..384445b1ae6 100755 --- a/tests/queries/0_stateless/00719_insert_block_without_column.sh +++ b/tests/queries/0_stateless/00719_insert_block_without_column.sh @@ -19,3 +19,5 @@ ${CLICKHOUSE_CLIENT} --query "create table squashed_numbers (SomeID UInt64, Diff ${CLICKHOUSE_CURL} -sS --data-binary "@${CLICKHOUSE_TMP}/test_squashing_block_without_column.out" "${CLICKHOUSE_URL}&query=insert%20into%20squashed_numbers%20format%20Native" ${CLICKHOUSE_CLIENT} --query "select 'Still alive'" + +${CLICKHOUSE_CLIENT} --query "drop table squashed_numbers" diff --git a/tests/queries/0_stateless/00754_alter_modify_order_by_replicated_zookeeper.reference b/tests/queries/0_stateless/00754_alter_modify_order_by_replicated_zookeeper.reference index 4a7d4e3ff00..48fe2d30bf3 100644 --- a/tests/queries/0_stateless/00754_alter_modify_order_by_replicated_zookeeper.reference +++ b/tests/queries/0_stateless/00754_alter_modify_order_by_replicated_zookeeper.reference @@ -9,6 +9,6 @@ 1 2 1 30 1 2 4 90 *** Check SHOW CREATE TABLE *** -CREATE TABLE test.summing_r2\n(\n `x` UInt32,\n `y` UInt32,\n `z` UInt32,\n `val` UInt32\n)\nENGINE = ReplicatedSummingMergeTree(\'/clickhouse/tables/test_00754/summing\', \'r2\')\nPRIMARY KEY (x, y)\nORDER BY (x, y, -z)\nSETTINGS index_granularity = 8192 +CREATE TABLE default.summing_r2\n(\n `x` UInt32,\n `y` UInt32,\n `z` UInt32,\n `val` UInt32\n)\nENGINE = ReplicatedSummingMergeTree(\'/clickhouse/tables/test_00754/summing\', \'r2\')\nPRIMARY KEY (x, y)\nORDER BY (x, y, -z)\nSETTINGS index_granularity = 8192 *** Check SHOW CREATE TABLE after offline ALTER *** -CREATE TABLE test.summing_r2\n(\n `x` UInt32,\n `y` UInt32,\n `z` UInt32,\n `t` UInt32,\n `val` UInt32\n)\nENGINE = ReplicatedSummingMergeTree(\'/clickhouse/tables/test_00754/summing\', \'r2\')\nPRIMARY KEY (x, y)\nORDER BY (x, y, t * t)\nSETTINGS index_granularity = 8192 +CREATE TABLE default.summing_r2\n(\n `x` UInt32,\n `y` UInt32,\n `z` UInt32,\n `t` UInt32,\n `val` UInt32\n)\nENGINE = ReplicatedSummingMergeTree(\'/clickhouse/tables/test_00754/summing\', \'r2\')\nPRIMARY KEY (x, y)\nORDER BY (x, y, t * t)\nSETTINGS index_granularity = 8192 diff --git a/tests/queries/0_stateless/00754_alter_modify_order_by_replicated_zookeeper.sql b/tests/queries/0_stateless/00754_alter_modify_order_by_replicated_zookeeper.sql index 0f861749537..4b150fb3826 100644 --- a/tests/queries/0_stateless/00754_alter_modify_order_by_replicated_zookeeper.sql +++ b/tests/queries/0_stateless/00754_alter_modify_order_by_replicated_zookeeper.sql @@ -5,48 +5,48 @@ CREATE TABLE old_style(d Date, x UInt32) ENGINE ReplicatedMergeTree('/clickhouse ALTER TABLE old_style ADD COLUMN y UInt32, MODIFY ORDER BY (x, y); -- { serverError 36 } DROP TABLE old_style; -DROP TABLE IF EXISTS test.summing_r1; -DROP TABLE IF EXISTS test.summing_r2; -CREATE TABLE test.summing_r1(x UInt32, y UInt32, val UInt32) ENGINE ReplicatedSummingMergeTree('/clickhouse/tables/test_00754/summing', 'r1') ORDER BY (x, y); -CREATE TABLE test.summing_r2(x UInt32, y UInt32, val UInt32) ENGINE ReplicatedSummingMergeTree('/clickhouse/tables/test_00754/summing', 'r2') ORDER BY (x, y); +DROP TABLE IF EXISTS summing_r1; +DROP TABLE IF EXISTS summing_r2; +CREATE TABLE summing_r1(x UInt32, y UInt32, val UInt32) ENGINE ReplicatedSummingMergeTree('/clickhouse/tables/test_00754/summing', 'r1') ORDER BY (x, y); +CREATE TABLE summing_r2(x UInt32, y UInt32, val UInt32) ENGINE ReplicatedSummingMergeTree('/clickhouse/tables/test_00754/summing', 'r2') ORDER BY (x, y); /* Can't add an expression with existing column to ORDER BY. */ -ALTER TABLE test.summing_r1 MODIFY ORDER BY (x, y, -val); -- { serverError 36 } +ALTER TABLE summing_r1 MODIFY ORDER BY (x, y, -val); -- { serverError 36 } /* Can't add an expression with existing column to ORDER BY. */ -ALTER TABLE test.summing_r1 ADD COLUMN z UInt32 DEFAULT x + 1, MODIFY ORDER BY (x, y, -z); -- { serverError 36 } +ALTER TABLE summing_r1 ADD COLUMN z UInt32 DEFAULT x + 1, MODIFY ORDER BY (x, y, -z); -- { serverError 36 } /* Can't add nonexistent column to ORDER BY. */ -ALTER TABLE test.summing_r1 MODIFY ORDER BY (x, y, nonexistent); -- { serverError 47 } +ALTER TABLE summing_r1 MODIFY ORDER BY (x, y, nonexistent); -- { serverError 47 } /* Can't modyfy ORDER BY so that it is no longer a prefix of the PRIMARY KEY. */ -ALTER TABLE test.summing_r1 MODIFY ORDER BY x; -- { serverError 36 } +ALTER TABLE summing_r1 MODIFY ORDER BY x; -- { serverError 36 } -INSERT INTO test.summing_r1(x, y, val) VALUES (1, 2, 10), (1, 2, 20); -SYSTEM SYNC REPLICA test.summing_r2; +INSERT INTO summing_r1(x, y, val) VALUES (1, 2, 10), (1, 2, 20); +SYSTEM SYNC REPLICA summing_r2; -ALTER TABLE test.summing_r1 ADD COLUMN z UInt32 AFTER y, MODIFY ORDER BY (x, y, -z); +ALTER TABLE summing_r1 ADD COLUMN z UInt32 AFTER y, MODIFY ORDER BY (x, y, -z); -INSERT INTO test.summing_r1(x, y, z, val) values (1, 2, 1, 30), (1, 2, 2, 40), (1, 2, 2, 50); -SYSTEM SYNC REPLICA test.summing_r2; +INSERT INTO summing_r1(x, y, z, val) values (1, 2, 1, 30), (1, 2, 2, 40), (1, 2, 2, 50); +SYSTEM SYNC REPLICA summing_r2; SELECT '*** Check that the parts are sorted according to the new key. ***'; -SELECT * FROM test.summing_r2 ORDER BY _part; +SELECT * FROM summing_r2 ORDER BY _part; SELECT '*** Check that the rows are collapsed according to the new key. ***'; -SELECT * FROM test.summing_r2 FINAL ORDER BY x, y, z; +SELECT * FROM summing_r2 FINAL ORDER BY x, y, z; SELECT '*** Check SHOW CREATE TABLE ***'; -SHOW CREATE TABLE test.summing_r2; +SHOW CREATE TABLE summing_r2; -DETACH TABLE test.summing_r2; -ALTER TABLE test.summing_r1 ADD COLUMN t UInt32 AFTER z, MODIFY ORDER BY (x, y, t * t) SETTINGS replication_alter_partitions_sync = 2; -- { serverError 341 } -ATTACH TABLE test.summing_r2; +DETACH TABLE summing_r2; +ALTER TABLE summing_r1 ADD COLUMN t UInt32 AFTER z, MODIFY ORDER BY (x, y, t * t) SETTINGS replication_alter_partitions_sync = 2; -- { serverError 341 } +ATTACH TABLE summing_r2; -SYSTEM SYNC REPLICA test.summing_r2; +SYSTEM SYNC REPLICA summing_r2; SELECT '*** Check SHOW CREATE TABLE after offline ALTER ***'; -SHOW CREATE TABLE test.summing_r2; +SHOW CREATE TABLE summing_r2; -DROP TABLE test.summing_r1; -DROP TABLE test.summing_r2; +DROP TABLE summing_r1; +DROP TABLE summing_r2; diff --git a/tests/queries/0_stateless/00836_indices_alter_replicated_zookeeper.reference b/tests/queries/0_stateless/00836_indices_alter_replicated_zookeeper.reference index 4029f0666b7..838bd93ebaf 100644 --- a/tests/queries/0_stateless/00836_indices_alter_replicated_zookeeper.reference +++ b/tests/queries/0_stateless/00836_indices_alter_replicated_zookeeper.reference @@ -1,5 +1,5 @@ -CREATE TABLE test.minmax_idx\n(\n `u64` UInt64,\n `i32` Int32,\n INDEX idx1 u64 * i32 TYPE minmax GRANULARITY 10,\n INDEX idx3 u64 - i32 TYPE minmax GRANULARITY 10,\n INDEX idx2 u64 + i32 TYPE minmax GRANULARITY 10\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00836/indices_alter1\', \'r1\')\nORDER BY u64\nSETTINGS index_granularity = 8192 -CREATE TABLE test.minmax_idx_r\n(\n `u64` UInt64,\n `i32` Int32,\n INDEX idx1 u64 * i32 TYPE minmax GRANULARITY 10,\n INDEX idx3 u64 - i32 TYPE minmax GRANULARITY 10,\n INDEX idx2 u64 + i32 TYPE minmax GRANULARITY 10\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00836/indices_alter1\', \'r2\')\nORDER BY u64\nSETTINGS index_granularity = 8192 +CREATE TABLE default.minmax_idx\n(\n `u64` UInt64,\n `i32` Int32,\n INDEX idx1 u64 * i32 TYPE minmax GRANULARITY 10,\n INDEX idx3 u64 - i32 TYPE minmax GRANULARITY 10,\n INDEX idx2 u64 + i32 TYPE minmax GRANULARITY 10\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00836/indices_alter1\', \'r1\')\nORDER BY u64\nSETTINGS index_granularity = 8192 +CREATE TABLE default.minmax_idx_r\n(\n `u64` UInt64,\n `i32` Int32,\n INDEX idx1 u64 * i32 TYPE minmax GRANULARITY 10,\n INDEX idx3 u64 - i32 TYPE minmax GRANULARITY 10,\n INDEX idx2 u64 + i32 TYPE minmax GRANULARITY 10\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00836/indices_alter1\', \'r2\')\nORDER BY u64\nSETTINGS index_granularity = 8192 1 2 1 2 1 2 @@ -14,8 +14,8 @@ CREATE TABLE test.minmax_idx_r\n(\n `u64` UInt64,\n `i32` Int32,\n INDE 3 2 19 9 65 75 -CREATE TABLE test.minmax_idx\n(\n `u64` UInt64,\n `i32` Int32,\n INDEX idx3 u64 - i32 TYPE minmax GRANULARITY 10,\n INDEX idx2 u64 + i32 TYPE minmax GRANULARITY 10\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00836/indices_alter1\', \'r1\')\nORDER BY u64\nSETTINGS index_granularity = 8192 -CREATE TABLE test.minmax_idx_r\n(\n `u64` UInt64,\n `i32` Int32,\n INDEX idx3 u64 - i32 TYPE minmax GRANULARITY 10,\n INDEX idx2 u64 + i32 TYPE minmax GRANULARITY 10\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00836/indices_alter1\', \'r2\')\nORDER BY u64\nSETTINGS index_granularity = 8192 +CREATE TABLE default.minmax_idx\n(\n `u64` UInt64,\n `i32` Int32,\n INDEX idx3 u64 - i32 TYPE minmax GRANULARITY 10,\n INDEX idx2 u64 + i32 TYPE minmax GRANULARITY 10\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00836/indices_alter1\', \'r1\')\nORDER BY u64\nSETTINGS index_granularity = 8192 +CREATE TABLE default.minmax_idx_r\n(\n `u64` UInt64,\n `i32` Int32,\n INDEX idx3 u64 - i32 TYPE minmax GRANULARITY 10,\n INDEX idx2 u64 + i32 TYPE minmax GRANULARITY 10\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00836/indices_alter1\', \'r2\')\nORDER BY u64\nSETTINGS index_granularity = 8192 1 2 1 4 1 5 @@ -28,10 +28,10 @@ CREATE TABLE test.minmax_idx_r\n(\n `u64` UInt64,\n `i32` Int32,\n INDE 3 2 19 9 65 75 -CREATE TABLE test.minmax_idx\n(\n `u64` UInt64,\n `i32` Int32\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00836/indices_alter1\', \'r1\')\nORDER BY u64\nSETTINGS index_granularity = 8192 -CREATE TABLE test.minmax_idx_r\n(\n `u64` UInt64,\n `i32` Int32\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00836/indices_alter1\', \'r2\')\nORDER BY u64\nSETTINGS index_granularity = 8192 -CREATE TABLE test.minmax_idx\n(\n `u64` UInt64,\n `i32` Int32,\n INDEX idx1 u64 * i32 TYPE minmax GRANULARITY 10\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00836/indices_alter1\', \'r1\')\nORDER BY u64\nSETTINGS index_granularity = 8192 -CREATE TABLE test.minmax_idx_r\n(\n `u64` UInt64,\n `i32` Int32,\n INDEX idx1 u64 * i32 TYPE minmax GRANULARITY 10\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00836/indices_alter1\', \'r2\')\nORDER BY u64\nSETTINGS index_granularity = 8192 +CREATE TABLE default.minmax_idx\n(\n `u64` UInt64,\n `i32` Int32\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00836/indices_alter1\', \'r1\')\nORDER BY u64\nSETTINGS index_granularity = 8192 +CREATE TABLE default.minmax_idx_r\n(\n `u64` UInt64,\n `i32` Int32\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00836/indices_alter1\', \'r2\')\nORDER BY u64\nSETTINGS index_granularity = 8192 +CREATE TABLE default.minmax_idx\n(\n `u64` UInt64,\n `i32` Int32,\n INDEX idx1 u64 * i32 TYPE minmax GRANULARITY 10\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00836/indices_alter1\', \'r1\')\nORDER BY u64\nSETTINGS index_granularity = 8192 +CREATE TABLE default.minmax_idx_r\n(\n `u64` UInt64,\n `i32` Int32,\n INDEX idx1 u64 * i32 TYPE minmax GRANULARITY 10\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00836/indices_alter1\', \'r2\')\nORDER BY u64\nSETTINGS index_granularity = 8192 1 2 1 4 1 5 @@ -44,14 +44,14 @@ CREATE TABLE test.minmax_idx_r\n(\n `u64` UInt64,\n `i32` Int32,\n INDE 3 2 19 9 65 75 -CREATE TABLE test.minmax_idx2\n(\n `u64` UInt64,\n `i32` Int32,\n INDEX idx1 u64 + i32 TYPE minmax GRANULARITY 10,\n INDEX idx2 u64 * i32 TYPE minmax GRANULARITY 10\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00836/indices_alter2\', \'r1\')\nORDER BY u64\nSETTINGS index_granularity = 8192 -CREATE TABLE test.minmax_idx2_r\n(\n `u64` UInt64,\n `i32` Int32,\n INDEX idx1 u64 + i32 TYPE minmax GRANULARITY 10,\n INDEX idx2 u64 * i32 TYPE minmax GRANULARITY 10\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00836/indices_alter2\', \'r2\')\nORDER BY u64\nSETTINGS index_granularity = 8192 +CREATE TABLE default.minmax_idx2\n(\n `u64` UInt64,\n `i32` Int32,\n INDEX idx1 u64 + i32 TYPE minmax GRANULARITY 10,\n INDEX idx2 u64 * i32 TYPE minmax GRANULARITY 10\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00836/indices_alter2\', \'r1\')\nORDER BY u64\nSETTINGS index_granularity = 8192 +CREATE TABLE default.minmax_idx2_r\n(\n `u64` UInt64,\n `i32` Int32,\n INDEX idx1 u64 + i32 TYPE minmax GRANULARITY 10,\n INDEX idx2 u64 * i32 TYPE minmax GRANULARITY 10\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00836/indices_alter2\', \'r2\')\nORDER BY u64\nSETTINGS index_granularity = 8192 1 2 1 3 1 2 1 3 -CREATE TABLE test.minmax_idx2\n(\n `u64` UInt64,\n `i32` Int32\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00836/indices_alter2\', \'r1\')\nORDER BY u64\nSETTINGS index_granularity = 8192 -CREATE TABLE test.minmax_idx2_r\n(\n `u64` UInt64,\n `i32` Int32\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00836/indices_alter2\', \'r2\')\nORDER BY u64\nSETTINGS index_granularity = 8192 +CREATE TABLE default.minmax_idx2\n(\n `u64` UInt64,\n `i32` Int32\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00836/indices_alter2\', \'r1\')\nORDER BY u64\nSETTINGS index_granularity = 8192 +CREATE TABLE default.minmax_idx2_r\n(\n `u64` UInt64,\n `i32` Int32\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00836/indices_alter2\', \'r2\')\nORDER BY u64\nSETTINGS index_granularity = 8192 1 2 1 3 1 2 diff --git a/tests/queries/0_stateless/00836_indices_alter_replicated_zookeeper.sql b/tests/queries/0_stateless/00836_indices_alter_replicated_zookeeper.sql index 5773357c675..e038d2d425e 100644 --- a/tests/queries/0_stateless/00836_indices_alter_replicated_zookeeper.sql +++ b/tests/queries/0_stateless/00836_indices_alter_replicated_zookeeper.sql @@ -1,74 +1,74 @@ -DROP TABLE IF EXISTS test.minmax_idx; -DROP TABLE IF EXISTS test.minmax_idx_r; -DROP TABLE IF EXISTS test.minmax_idx2; -DROP TABLE IF EXISTS test.minmax_idx2_r; +DROP TABLE IF EXISTS minmax_idx; +DROP TABLE IF EXISTS minmax_idx_r; +DROP TABLE IF EXISTS minmax_idx2; +DROP TABLE IF EXISTS minmax_idx2_r; SET replication_alter_partitions_sync = 2; -CREATE TABLE test.minmax_idx +CREATE TABLE minmax_idx ( u64 UInt64, i32 Int32 ) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test_00836/indices_alter1', 'r1') ORDER BY u64; -CREATE TABLE test.minmax_idx_r +CREATE TABLE minmax_idx_r ( u64 UInt64, i32 Int32 ) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test_00836/indices_alter1', 'r2') ORDER BY u64; -INSERT INTO test.minmax_idx VALUES (1, 2); +INSERT INTO minmax_idx VALUES (1, 2); -SYSTEM SYNC REPLICA test.minmax_idx_r; +SYSTEM SYNC REPLICA minmax_idx_r; -ALTER TABLE test.minmax_idx ADD INDEX idx1 u64 * i32 TYPE minmax GRANULARITY 10; -ALTER TABLE test.minmax_idx_r ADD INDEX idx2 u64 + i32 TYPE minmax GRANULARITY 10; -ALTER TABLE test.minmax_idx ADD INDEX idx3 u64 - i32 TYPE minmax GRANULARITY 10 AFTER idx1; +ALTER TABLE minmax_idx ADD INDEX idx1 u64 * i32 TYPE minmax GRANULARITY 10; +ALTER TABLE minmax_idx_r ADD INDEX idx2 u64 + i32 TYPE minmax GRANULARITY 10; +ALTER TABLE minmax_idx ADD INDEX idx3 u64 - i32 TYPE minmax GRANULARITY 10 AFTER idx1; -SHOW CREATE TABLE test.minmax_idx; -SHOW CREATE TABLE test.minmax_idx_r; +SHOW CREATE TABLE minmax_idx; +SHOW CREATE TABLE minmax_idx_r; -SELECT * FROM test.minmax_idx WHERE u64 * i32 = 2 ORDER BY (u64, i32); -SELECT * FROM test.minmax_idx_r WHERE u64 * i32 = 2 ORDER BY (u64, i32); +SELECT * FROM minmax_idx WHERE u64 * i32 = 2 ORDER BY (u64, i32); +SELECT * FROM minmax_idx_r WHERE u64 * i32 = 2 ORDER BY (u64, i32); -INSERT INTO test.minmax_idx VALUES (1, 4); -INSERT INTO test.minmax_idx_r VALUES (3, 2); -INSERT INTO test.minmax_idx VALUES (1, 5); -INSERT INTO test.minmax_idx_r VALUES (65, 75); -INSERT INTO test.minmax_idx VALUES (19, 9); +INSERT INTO minmax_idx VALUES (1, 4); +INSERT INTO minmax_idx_r VALUES (3, 2); +INSERT INTO minmax_idx VALUES (1, 5); +INSERT INTO minmax_idx_r VALUES (65, 75); +INSERT INTO minmax_idx VALUES (19, 9); -SYSTEM SYNC REPLICA test.minmax_idx; -SYSTEM SYNC REPLICA test.minmax_idx_r; +SYSTEM SYNC REPLICA minmax_idx; +SYSTEM SYNC REPLICA minmax_idx_r; -SELECT * FROM test.minmax_idx WHERE u64 * i32 > 1 ORDER BY (u64, i32); -SELECT * FROM test.minmax_idx_r WHERE u64 * i32 > 1 ORDER BY (u64, i32); +SELECT * FROM minmax_idx WHERE u64 * i32 > 1 ORDER BY (u64, i32); +SELECT * FROM minmax_idx_r WHERE u64 * i32 > 1 ORDER BY (u64, i32); -ALTER TABLE test.minmax_idx DROP INDEX idx1; +ALTER TABLE minmax_idx DROP INDEX idx1; -SHOW CREATE TABLE test.minmax_idx; -SHOW CREATE TABLE test.minmax_idx_r; +SHOW CREATE TABLE minmax_idx; +SHOW CREATE TABLE minmax_idx_r; -SELECT * FROM test.minmax_idx WHERE u64 * i32 > 1 ORDER BY (u64, i32); -SELECT * FROM test.minmax_idx_r WHERE u64 * i32 > 1 ORDER BY (u64, i32); +SELECT * FROM minmax_idx WHERE u64 * i32 > 1 ORDER BY (u64, i32); +SELECT * FROM minmax_idx_r WHERE u64 * i32 > 1 ORDER BY (u64, i32); -ALTER TABLE test.minmax_idx DROP INDEX idx2; -ALTER TABLE test.minmax_idx_r DROP INDEX idx3; +ALTER TABLE minmax_idx DROP INDEX idx2; +ALTER TABLE minmax_idx_r DROP INDEX idx3; -SHOW CREATE TABLE test.minmax_idx; -SHOW CREATE TABLE test.minmax_idx_r; +SHOW CREATE TABLE minmax_idx; +SHOW CREATE TABLE minmax_idx_r; -ALTER TABLE test.minmax_idx ADD INDEX idx1 u64 * i32 TYPE minmax GRANULARITY 10; +ALTER TABLE minmax_idx ADD INDEX idx1 u64 * i32 TYPE minmax GRANULARITY 10; -SHOW CREATE TABLE test.minmax_idx; -SHOW CREATE TABLE test.minmax_idx_r; +SHOW CREATE TABLE minmax_idx; +SHOW CREATE TABLE minmax_idx_r; -SELECT * FROM test.minmax_idx WHERE u64 * i32 > 1 ORDER BY (u64, i32); -SELECT * FROM test.minmax_idx_r WHERE u64 * i32 > 1 ORDER BY (u64, i32); +SELECT * FROM minmax_idx WHERE u64 * i32 > 1 ORDER BY (u64, i32); +SELECT * FROM minmax_idx_r WHERE u64 * i32 > 1 ORDER BY (u64, i32); -CREATE TABLE test.minmax_idx2 +CREATE TABLE minmax_idx2 ( u64 UInt64, i32 Int32, @@ -77,7 +77,7 @@ CREATE TABLE test.minmax_idx2 ) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test_00836/indices_alter2', 'r1') ORDER BY u64; -CREATE TABLE test.minmax_idx2_r +CREATE TABLE minmax_idx2_r ( u64 UInt64, i32 Int32, @@ -87,27 +87,27 @@ CREATE TABLE test.minmax_idx2_r ORDER BY u64; -SHOW CREATE TABLE test.minmax_idx2; -SHOW CREATE TABLE test.minmax_idx2_r; +SHOW CREATE TABLE minmax_idx2; +SHOW CREATE TABLE minmax_idx2_r; -INSERT INTO test.minmax_idx2 VALUES (1, 2); -INSERT INTO test.minmax_idx2_r VALUES (1, 3); +INSERT INTO minmax_idx2 VALUES (1, 2); +INSERT INTO minmax_idx2_r VALUES (1, 3); -SYSTEM SYNC REPLICA test.minmax_idx2; -SYSTEM SYNC REPLICA test.minmax_idx2_r; +SYSTEM SYNC REPLICA minmax_idx2; +SYSTEM SYNC REPLICA minmax_idx2_r; -SELECT * FROM test.minmax_idx2 WHERE u64 * i32 >= 2 ORDER BY (u64, i32); -SELECT * FROM test.minmax_idx2_r WHERE u64 * i32 >= 2 ORDER BY (u64, i32); +SELECT * FROM minmax_idx2 WHERE u64 * i32 >= 2 ORDER BY (u64, i32); +SELECT * FROM minmax_idx2_r WHERE u64 * i32 >= 2 ORDER BY (u64, i32); -ALTER TABLE test.minmax_idx2_r DROP INDEX idx1, DROP INDEX idx2; +ALTER TABLE minmax_idx2_r DROP INDEX idx1, DROP INDEX idx2; -SHOW CREATE TABLE test.minmax_idx2; -SHOW CREATE TABLE test.minmax_idx2_r; +SHOW CREATE TABLE minmax_idx2; +SHOW CREATE TABLE minmax_idx2_r; -SELECT * FROM test.minmax_idx2 WHERE u64 * i32 >= 2 ORDER BY (u64, i32); -SELECT * FROM test.minmax_idx2_r WHERE u64 * i32 >= 2 ORDER BY (u64, i32); +SELECT * FROM minmax_idx2 WHERE u64 * i32 >= 2 ORDER BY (u64, i32); +SELECT * FROM minmax_idx2_r WHERE u64 * i32 >= 2 ORDER BY (u64, i32); -DROP TABLE test.minmax_idx; -DROP TABLE test.minmax_idx_r; -DROP TABLE test.minmax_idx2; -DROP TABLE test.minmax_idx2_r; +DROP TABLE minmax_idx; +DROP TABLE minmax_idx_r; +DROP TABLE minmax_idx2; +DROP TABLE minmax_idx2_r; diff --git a/tests/queries/0_stateless/00900_orc_load.sh b/tests/queries/0_stateless/00900_orc_load.sh index 5d2e45e2574..6e08b415397 100755 --- a/tests/queries/0_stateless/00900_orc_load.sh +++ b/tests/queries/0_stateless/00900_orc_load.sh @@ -10,3 +10,4 @@ ${CLICKHOUSE_CLIENT} --query="CREATE TABLE orc_load (int Int32, smallint Int8, b cat "$DATA_FILE" | ${CLICKHOUSE_CLIENT} -q "insert into orc_load format ORC" ${CLICKHOUSE_CLIENT} --query="select * from orc_load" +${CLICKHOUSE_CLIENT} --query="drop table orc_load" diff --git a/tests/queries/0_stateless/00910_zookeeper_custom_compression_codecs_replicated.reference b/tests/queries/0_stateless/00910_zookeeper_custom_compression_codecs_replicated.reference index 328022414b8..29e7b23d3dd 100644 --- a/tests/queries/0_stateless/00910_zookeeper_custom_compression_codecs_replicated.reference +++ b/tests/queries/0_stateless/00910_zookeeper_custom_compression_codecs_replicated.reference @@ -20,7 +20,7 @@ 274972506.6 9175437371954010821 9175437371954010821 -CREATE TABLE test.compression_codec_multiple_more_types_replicated\n(\n `id` Decimal(38, 13) CODEC(ZSTD(1), LZ4, ZSTD(1), ZSTD(1), Delta(2), Delta(4), Delta(1), LZ4HC(0)),\n `data` FixedString(12) CODEC(ZSTD(1), ZSTD(1), Delta(1), Delta(1), Delta(1), NONE, NONE, NONE, LZ4HC(0)),\n `ddd.age` Array(UInt8) CODEC(LZ4, LZ4HC(0), NONE, NONE, NONE, ZSTD(1), Delta(8)),\n `ddd.Name` Array(String) CODEC(LZ4, LZ4HC(0), NONE, NONE, NONE, ZSTD(1), Delta(8))\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00910/compression_codec_multiple_more_types_replicated\', \'1\')\nORDER BY tuple()\nSETTINGS index_granularity = 8192 +CREATE TABLE default.compression_codec_multiple_more_types_replicated\n(\n `id` Decimal(38, 13) CODEC(ZSTD(1), LZ4, ZSTD(1), ZSTD(1), Delta(2), Delta(4), Delta(1), LZ4HC(0)),\n `data` FixedString(12) CODEC(ZSTD(1), ZSTD(1), Delta(1), Delta(1), Delta(1), NONE, NONE, NONE, LZ4HC(0)),\n `ddd.age` Array(UInt8) CODEC(LZ4, LZ4HC(0), NONE, NONE, NONE, ZSTD(1), Delta(8)),\n `ddd.Name` Array(String) CODEC(LZ4, LZ4HC(0), NONE, NONE, NONE, ZSTD(1), Delta(8))\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00910/compression_codec_multiple_more_types_replicated\', \'1\')\nORDER BY tuple()\nSETTINGS index_granularity = 8192 1.5555555555555 hello world! [77] ['John'] 7.1000000000000 xxxxxxxxxxxx [127] ['Henry'] ! diff --git a/tests/queries/0_stateless/00910_zookeeper_custom_compression_codecs_replicated.sql b/tests/queries/0_stateless/00910_zookeeper_custom_compression_codecs_replicated.sql index e4c9f0ffc3b..52eb1d4e411 100644 --- a/tests/queries/0_stateless/00910_zookeeper_custom_compression_codecs_replicated.sql +++ b/tests/queries/0_stateless/00910_zookeeper_custom_compression_codecs_replicated.sql @@ -1,10 +1,10 @@ SET send_logs_level = 'fatal'; SET allow_suspicious_codecs = 1; -DROP TABLE IF EXISTS test.compression_codec_replicated1; -DROP TABLE IF EXISTS test.compression_codec_replicated2; +DROP TABLE IF EXISTS compression_codec_replicated1; +DROP TABLE IF EXISTS compression_codec_replicated2; -CREATE TABLE test.compression_codec_replicated1( +CREATE TABLE compression_codec_replicated1( id UInt64 CODEC(LZ4), data String CODEC(ZSTD), ddd Date CODEC(NONE), @@ -13,7 +13,7 @@ CREATE TABLE test.compression_codec_replicated1( othernum Int64 CODEC(Delta) ) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test_00910/compression_codec_replicated', '1') ORDER BY tuple(); -CREATE TABLE test.compression_codec_replicated2( +CREATE TABLE compression_codec_replicated2( id UInt64 CODEC(LZ4), data String CODEC(ZSTD), ddd Date CODEC(NONE), @@ -23,43 +23,43 @@ CREATE TABLE test.compression_codec_replicated2( ) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test_00910/compression_codec_replicated', '2') ORDER BY tuple(); -INSERT INTO test.compression_codec_replicated1 VALUES(1, 'hello', toDate('2018-12-14'), 1.1, 'aaa', 5); -INSERT INTO test.compression_codec_replicated1 VALUES(2, 'world', toDate('2018-12-15'), 2.2, 'bbb', 6); -INSERT INTO test.compression_codec_replicated1 VALUES(3, '!', toDate('2018-12-16'), 3.3, 'ccc', 7); +INSERT INTO compression_codec_replicated1 VALUES(1, 'hello', toDate('2018-12-14'), 1.1, 'aaa', 5); +INSERT INTO compression_codec_replicated1 VALUES(2, 'world', toDate('2018-12-15'), 2.2, 'bbb', 6); +INSERT INTO compression_codec_replicated1 VALUES(3, '!', toDate('2018-12-16'), 3.3, 'ccc', 7); -SYSTEM SYNC REPLICA test.compression_codec_replicated2; +SYSTEM SYNC REPLICA compression_codec_replicated2; -SELECT * FROM test.compression_codec_replicated1 ORDER BY id; -SELECT * FROM test.compression_codec_replicated2 ORDER BY id; +SELECT * FROM compression_codec_replicated1 ORDER BY id; +SELECT * FROM compression_codec_replicated2 ORDER BY id; -OPTIMIZE TABLE test.compression_codec_replicated1 FINAL; +OPTIMIZE TABLE compression_codec_replicated1 FINAL; -INSERT INTO test.compression_codec_replicated1 VALUES(2, '', toDate('2018-12-13'), 4.4, 'ddd', 8); +INSERT INTO compression_codec_replicated1 VALUES(2, '', toDate('2018-12-13'), 4.4, 'ddd', 8); -SYSTEM SYNC REPLICA test.compression_codec_replicated2; +SYSTEM SYNC REPLICA compression_codec_replicated2; -DETACH TABLE test.compression_codec_replicated1; -ATTACH TABLE test.compression_codec_replicated1; +DETACH TABLE compression_codec_replicated1; +ATTACH TABLE compression_codec_replicated1; -SELECT count(*) FROM test.compression_codec_replicated1 WHERE id = 2 GROUP BY id; -SELECT count(*) FROM test.compression_codec_replicated2 WHERE id = 2 GROUP BY id; +SELECT count(*) FROM compression_codec_replicated1 WHERE id = 2 GROUP BY id; +SELECT count(*) FROM compression_codec_replicated2 WHERE id = 2 GROUP BY id; -DROP TABLE IF EXISTS test.compression_codec_replicated1; -DROP TABLE IF EXISTS test.compression_codec_replicated2; +DROP TABLE IF EXISTS compression_codec_replicated1; +DROP TABLE IF EXISTS compression_codec_replicated2; -DROP TABLE IF EXISTS test.compression_codec_multiple_replicated1; -DROP TABLE IF EXISTS test.compression_codec_multiple_replicated2; +DROP TABLE IF EXISTS compression_codec_multiple_replicated1; +DROP TABLE IF EXISTS compression_codec_multiple_replicated2; SET network_compression_method = 'lz4hc'; -CREATE TABLE test.compression_codec_multiple_replicated1 ( +CREATE TABLE compression_codec_multiple_replicated1 ( id UInt64 CODEC(LZ4, ZSTD, NONE, LZ4HC, Delta(4)), data String CODEC(ZSTD(2), NONE, Delta(2), LZ4HC, LZ4, LZ4, Delta(8)), ddd Date CODEC(NONE, NONE, NONE, Delta(1), LZ4, ZSTD, LZ4HC, LZ4HC), somenum Float64 CODEC(Delta(4), LZ4, LZ4, ZSTD(2), LZ4HC(5), ZSTD(3), ZSTD) ) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test_00910/compression_codec_multiple', '1') ORDER BY tuple(); -CREATE TABLE test.compression_codec_multiple_replicated2 ( +CREATE TABLE compression_codec_multiple_replicated2 ( id UInt64 CODEC(LZ4, ZSTD, NONE, LZ4HC, Delta(4)), data String CODEC(ZSTD(2), NONE, Delta(2), LZ4HC, LZ4, LZ4, Delta(8)), ddd Date CODEC(NONE, NONE, NONE, Delta(1), LZ4, ZSTD, LZ4HC, LZ4HC), @@ -67,72 +67,73 @@ CREATE TABLE test.compression_codec_multiple_replicated2 ( ) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test_00910/compression_codec_multiple', '2') ORDER BY tuple(); -INSERT INTO test.compression_codec_multiple_replicated2 VALUES (1, 'world', toDate('2018-10-05'), 1.1), (2, 'hello', toDate('2018-10-01'), 2.2), (3, 'buy', toDate('2018-10-11'), 3.3); +INSERT INTO compression_codec_multiple_replicated2 VALUES (1, 'world', toDate('2018-10-05'), 1.1), (2, 'hello', toDate('2018-10-01'), 2.2), (3, 'buy', toDate('2018-10-11'), 3.3); -SYSTEM SYNC REPLICA test.compression_codec_multiple_replicated1; +SYSTEM SYNC REPLICA compression_codec_multiple_replicated1; -SELECT * FROM test.compression_codec_multiple_replicated2 ORDER BY id; -SELECT * FROM test.compression_codec_multiple_replicated1 ORDER BY id; +SELECT * FROM compression_codec_multiple_replicated2 ORDER BY id; +SELECT * FROM compression_codec_multiple_replicated1 ORDER BY id; -INSERT INTO test.compression_codec_multiple_replicated1 select modulo(number, 100), toString(number), toDate('2018-12-01'), 5.5 * number FROM system.numbers limit 10000; +INSERT INTO compression_codec_multiple_replicated1 select modulo(number, 100), toString(number), toDate('2018-12-01'), 5.5 * number FROM system.numbers limit 10000; -SYSTEM SYNC REPLICA test.compression_codec_multiple_replicated2; +SYSTEM SYNC REPLICA compression_codec_multiple_replicated2; -SELECT count(*) FROM test.compression_codec_multiple_replicated1; -SELECT count(*) FROM test.compression_codec_multiple_replicated2; +SELECT count(*) FROM compression_codec_multiple_replicated1; +SELECT count(*) FROM compression_codec_multiple_replicated2; -SELECT count(distinct data) FROM test.compression_codec_multiple_replicated1; -SELECT count(distinct data) FROM test.compression_codec_multiple_replicated2; +SELECT count(distinct data) FROM compression_codec_multiple_replicated1; +SELECT count(distinct data) FROM compression_codec_multiple_replicated2; -SELECT floor(sum(somenum), 1) FROM test.compression_codec_multiple_replicated1; -SELECT floor(sum(somenum), 1) FROM test.compression_codec_multiple_replicated2; +SELECT floor(sum(somenum), 1) FROM compression_codec_multiple_replicated1; +SELECT floor(sum(somenum), 1) FROM compression_codec_multiple_replicated2; -TRUNCATE TABLE test.compression_codec_multiple_replicated1; -SYSTEM SYNC REPLICA test.compression_codec_multiple_replicated2; +TRUNCATE TABLE compression_codec_multiple_replicated1; +SYSTEM SYNC REPLICA compression_codec_multiple_replicated2; -INSERT INTO test.compression_codec_multiple_replicated1 select modulo(number, 100), toString(number), toDate('2018-12-01'), 5.5 * number FROM system.numbers limit 10000; +INSERT INTO compression_codec_multiple_replicated1 select modulo(number, 100), toString(number), toDate('2018-12-01'), 5.5 * number FROM system.numbers limit 10000; -SYSTEM SYNC REPLICA test.compression_codec_multiple_replicated2; +SYSTEM SYNC REPLICA compression_codec_multiple_replicated2; -SELECT sum(cityHash64(*)) FROM test.compression_codec_multiple_replicated2; -SELECT sum(cityHash64(*)) FROM test.compression_codec_multiple_replicated1; +SELECT sum(cityHash64(*)) FROM compression_codec_multiple_replicated2; +SELECT sum(cityHash64(*)) FROM compression_codec_multiple_replicated1; -DROP TABLE IF EXISTS test.compression_codec_multiple_replicated1; -DROP TABLE IF EXISTS test.compression_codec_multiple_replicated2; +DROP TABLE IF EXISTS compression_codec_multiple_replicated1; +DROP TABLE IF EXISTS compression_codec_multiple_replicated2; -DROP TABLE IF EXISTS test.compression_codec_multiple_more_types_replicated; +DROP TABLE IF EXISTS compression_codec_multiple_more_types_replicated; -CREATE TABLE test.compression_codec_multiple_more_types_replicated ( +CREATE TABLE compression_codec_multiple_more_types_replicated ( id Decimal128(13) CODEC(ZSTD, LZ4, ZSTD, ZSTD, Delta(2), Delta(4), Delta(1), LZ4HC), data FixedString(12) CODEC(ZSTD, ZSTD, Delta(1), Delta(1), Delta(1), NONE, NONE, NONE, LZ4HC), ddd Nested (age UInt8, Name String) CODEC(LZ4, LZ4HC, NONE, NONE, NONE, ZSTD, Delta(8)) ) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test_00910/compression_codec_multiple_more_types_replicated', '1') ORDER BY tuple(); -SHOW CREATE TABLE test.compression_codec_multiple_more_types_replicated; +SHOW CREATE TABLE compression_codec_multiple_more_types_replicated; -INSERT INTO test.compression_codec_multiple_more_types_replicated VALUES(1.5555555555555, 'hello world!', [77], ['John']); -INSERT INTO test.compression_codec_multiple_more_types_replicated VALUES(7.1, 'xxxxxxxxxxxx', [127], ['Henry']); +INSERT INTO compression_codec_multiple_more_types_replicated VALUES(1.5555555555555, 'hello world!', [77], ['John']); +INSERT INTO compression_codec_multiple_more_types_replicated VALUES(7.1, 'xxxxxxxxxxxx', [127], ['Henry']); -SELECT * FROM test.compression_codec_multiple_more_types_replicated order by id; +SELECT * FROM compression_codec_multiple_more_types_replicated order by id; -DROP TABLE IF EXISTS test.compression_codec_multiple_with_key_replicated; +DROP TABLE IF EXISTS compression_codec_multiple_with_key_replicated; SET network_compression_method = 'zstd'; SET network_zstd_compression_level = 5; -CREATE TABLE test.compression_codec_multiple_with_key_replicated ( +CREATE TABLE compression_codec_multiple_with_key_replicated ( somedate Date CODEC(ZSTD, ZSTD, ZSTD(12), LZ4HC(12), Delta, Delta), id UInt64 CODEC(LZ4, ZSTD, Delta, NONE, LZ4HC, Delta), data String CODEC(ZSTD(2), Delta(1), LZ4HC, NONE, LZ4, LZ4) ) ENGINE = ReplicatedMergeTree('/clickhouse/tables/test_00910/compression_codec_multiple_with_key_replicated', '1') PARTITION BY somedate ORDER BY id SETTINGS index_granularity = 2; -INSERT INTO test.compression_codec_multiple_with_key_replicated VALUES(toDate('2018-10-12'), 100000, 'hello'), (toDate('2018-10-12'), 100002, 'world'), (toDate('2018-10-12'), 1111, '!'); +INSERT INTO compression_codec_multiple_with_key_replicated VALUES(toDate('2018-10-12'), 100000, 'hello'), (toDate('2018-10-12'), 100002, 'world'), (toDate('2018-10-12'), 1111, '!'); -SELECT data FROM test.compression_codec_multiple_with_key_replicated WHERE id BETWEEN 3 AND 1112; +SELECT data FROM compression_codec_multiple_with_key_replicated WHERE id BETWEEN 3 AND 1112; -INSERT INTO test.compression_codec_multiple_with_key_replicated SELECT toDate('2018-10-12'), number, toString(number) FROM system.numbers LIMIT 1000; +INSERT INTO compression_codec_multiple_with_key_replicated SELECT toDate('2018-10-12'), number, toString(number) FROM system.numbers LIMIT 1000; -SELECT COUNT(DISTINCT data) FROM test.compression_codec_multiple_with_key_replicated WHERE id < 222; +SELECT COUNT(DISTINCT data) FROM compression_codec_multiple_with_key_replicated WHERE id < 222; -DROP TABLE IF EXISTS test.compression_codec_multiple_with_key_replicated; +DROP TABLE IF EXISTS compression_codec_multiple_with_key_replicated; +DROP TABLE compression_codec_multiple_more_types_replicated; diff --git a/tests/queries/0_stateless/00933_ttl_replicated_zookeeper.reference b/tests/queries/0_stateless/00933_ttl_replicated_zookeeper.reference index 16818d9c7dc..e84c3beabee 100644 --- a/tests/queries/0_stateless/00933_ttl_replicated_zookeeper.reference +++ b/tests/queries/0_stateless/00933_ttl_replicated_zookeeper.reference @@ -1,3 +1,3 @@ 200 400 -CREATE TABLE test.ttl_repl2\n(\n `d` Date,\n `x` UInt32\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00933/ttl_repl\', \'2\')\nPARTITION BY toDayOfMonth(d)\nORDER BY x\nTTL d + toIntervalDay(1)\nSETTINGS index_granularity = 8192 +CREATE TABLE default.ttl_repl2\n(\n `d` Date,\n `x` UInt32\n)\nENGINE = ReplicatedMergeTree(\'/clickhouse/tables/test_00933/ttl_repl\', \'2\')\nPARTITION BY toDayOfMonth(d)\nORDER BY x\nTTL d + toIntervalDay(1)\nSETTINGS index_granularity = 8192 diff --git a/tests/queries/0_stateless/00933_ttl_replicated_zookeeper.sql b/tests/queries/0_stateless/00933_ttl_replicated_zookeeper.sql index ac45f2903d5..dbbbe887e9f 100644 --- a/tests/queries/0_stateless/00933_ttl_replicated_zookeeper.sql +++ b/tests/queries/0_stateless/00933_ttl_replicated_zookeeper.sql @@ -1,25 +1,28 @@ -DROP TABLE IF EXISTS test.ttl_repl1; -DROP TABLE IF EXISTS test.ttl_repl2; +DROP TABLE IF EXISTS ttl_repl1; +DROP TABLE IF EXISTS ttl_repl2; -CREATE TABLE test.ttl_repl1(d Date, x UInt32) ENGINE ReplicatedMergeTree('/clickhouse/tables/test_00933/ttl_repl', '1') +CREATE TABLE ttl_repl1(d Date, x UInt32) ENGINE ReplicatedMergeTree('/clickhouse/tables/test_00933/ttl_repl', '1') PARTITION BY toDayOfMonth(d) ORDER BY x TTL d + INTERVAL 1 DAY; -CREATE TABLE test.ttl_repl2(d Date, x UInt32) ENGINE ReplicatedMergeTree('/clickhouse/tables/test_00933/ttl_repl', '2') +CREATE TABLE ttl_repl2(d Date, x UInt32) ENGINE ReplicatedMergeTree('/clickhouse/tables/test_00933/ttl_repl', '2') PARTITION BY toDayOfMonth(d) ORDER BY x TTL d + INTERVAL 1 DAY; -INSERT INTO TABLE test.ttl_repl1 VALUES (toDate('2000-10-10 00:00:00'), 100); -INSERT INTO TABLE test.ttl_repl1 VALUES (toDate('2100-10-10 00:00:00'), 200); +INSERT INTO TABLE ttl_repl1 VALUES (toDate('2000-10-10 00:00:00'), 100); +INSERT INTO TABLE ttl_repl1 VALUES (toDate('2100-10-10 00:00:00'), 200); -ALTER TABLE test.ttl_repl1 MODIFY TTL d + INTERVAL 1 DAY; -SYSTEM SYNC REPLICA test.ttl_repl2; +ALTER TABLE ttl_repl1 MODIFY TTL d + INTERVAL 1 DAY; +SYSTEM SYNC REPLICA ttl_repl2; -INSERT INTO TABLE test.ttl_repl1 VALUES (toDate('2000-10-10 00:00:00'), 300); -INSERT INTO TABLE test.ttl_repl1 VALUES (toDate('2100-10-10 00:00:00'), 400); +INSERT INTO TABLE ttl_repl1 VALUES (toDate('2000-10-10 00:00:00'), 300); +INSERT INTO TABLE ttl_repl1 VALUES (toDate('2100-10-10 00:00:00'), 400); -SYSTEM SYNC REPLICA test.ttl_repl2; +SYSTEM SYNC REPLICA ttl_repl2; SELECT sleep(1) format Null; -- wait for probable merges after inserts -OPTIMIZE TABLE test.ttl_repl2 FINAL; -SELECT x FROM test.ttl_repl2 ORDER BY x; +OPTIMIZE TABLE ttl_repl2 FINAL; +SELECT x FROM ttl_repl2 ORDER BY x; -SHOW CREATE TABLE test.ttl_repl2; +SHOW CREATE TABLE ttl_repl2; + +DROP TABLE ttl_repl1; +DROP TABLE ttl_repl2; diff --git a/tests/queries/0_stateless/00967_insert_into_distributed_different_types.sql b/tests/queries/0_stateless/00967_insert_into_distributed_different_types.sql index 33f16eb241c..455fab694cd 100644 --- a/tests/queries/0_stateless/00967_insert_into_distributed_different_types.sql +++ b/tests/queries/0_stateless/00967_insert_into_distributed_different_types.sql @@ -7,3 +7,6 @@ CREATE TABLE underlying_00967 (key Nullable(UInt64)) Engine=TinyLog(); INSERT INTO dist_00967 SELECT toUInt64(number) FROM system.numbers LIMIT 1; SELECT * FROM dist_00967; + +DROP TABLE dist_00967; +DROP TABLE underlying_00967; diff --git a/tests/queries/0_stateless/01056_create_table_as.sql b/tests/queries/0_stateless/01056_create_table_as.sql index 089daefe38d..32a39646170 100644 --- a/tests/queries/0_stateless/01056_create_table_as.sql +++ b/tests/queries/0_stateless/01056_create_table_as.sql @@ -45,3 +45,7 @@ CREATE TABLE t3 AS dict; -- { serverError 80; } DROP TABLE IF EXISTS t1; DROP TABLE IF EXISTS t3; +DROP DICTIONARY dict; +DROP TABLE test_01056_dict_data.dict_data; + +DROP DATABASE test_01056_dict_data; diff --git a/tests/queries/0_stateless/01059_storage_file_brotli.reference b/tests/queries/0_stateless/01059_storage_file_brotli.reference index 6c545e9faec..266e67c1445 100644 --- a/tests/queries/0_stateless/01059_storage_file_brotli.reference +++ b/tests/queries/0_stateless/01059_storage_file_brotli.reference @@ -1,5 +1,7 @@ 1000000 999999 1000000 999999 -2000000 999999 +1000000 999999 +3000000 999999 +1 255 1 255 1 255 diff --git a/tests/queries/0_stateless/01059_storage_file_brotli.sql b/tests/queries/0_stateless/01059_storage_file_brotli.sql index e7d5a87b2af..eba61e4450f 100644 --- a/tests/queries/0_stateless/01059_storage_file_brotli.sql +++ b/tests/queries/0_stateless/01059_storage_file_brotli.sql @@ -15,8 +15,17 @@ SELECT count(), max(x) FROM file; DROP TABLE file; -SELECT count(), max(x) FROM file('data{1,2}.tsv.{gz,br}', TSV, 'x UInt64'); +CREATE TABLE file (x UInt64) ENGINE = File(TSV, 'data3.tsv.xz'); +TRUNCATE TABLE file; + +INSERT INTO file SELECT * FROM numbers(1000000); +SELECT count(), max(x) FROM file; + +DROP TABLE file; + +SELECT count(), max(x) FROM file('data{1,2,3}.tsv.{gz,br,xz}', TSV, 'x UInt64'); -- check that they are compressed SELECT count() < 1000000, max(x) FROM file('data1.tsv.br', RowBinary, 'x UInt8', 'none'); SELECT count() < 3000000, max(x) FROM file('data2.tsv.gz', RowBinary, 'x UInt8', 'none'); +SELECT count() < 1000000, max(x) FROM file('data3.tsv.xz', RowBinary, 'x UInt8', 'none'); diff --git a/tests/queries/0_stateless/01114_materialize_clear_index_compact_parts.sql b/tests/queries/0_stateless/01114_materialize_clear_index_compact_parts.sql index 404922f36bb..831cb25d967 100644 --- a/tests/queries/0_stateless/01114_materialize_clear_index_compact_parts.sql +++ b/tests/queries/0_stateless/01114_materialize_clear_index_compact_parts.sql @@ -30,3 +30,5 @@ SELECT count() FROM minmax_compact WHERE i64 = 2; -- { serverError 158 } set max_rows_to_read = 10; SELECT count() FROM minmax_compact WHERE i64 = 2; + +DROP TABLE minmax_compact; diff --git a/tests/queries/0_stateless/01142_merge_join_lc_and_nullable_in_key.sql b/tests/queries/0_stateless/01142_merge_join_lc_and_nullable_in_key.sql index 8a1601e3faa..dbc2d7c9f5d 100644 --- a/tests/queries/0_stateless/01142_merge_join_lc_and_nullable_in_key.sql +++ b/tests/queries/0_stateless/01142_merge_join_lc_and_nullable_in_key.sql @@ -46,3 +46,6 @@ SELECT '-'; SELECT x, lc, materialize(r.lc) y, toTypeName(y) FROM t AS l LEFT JOIN nr AS r USING (lc) ORDER BY x; SELECT x, lc, materialize(r.lc) y, toTypeName(y) FROM t AS l RIGHT JOIN nr AS r USING (lc) ORDER BY x; SELECT x, lc, materialize(r.lc) y, toTypeName(y) FROM t AS l FULL JOIN nr AS r USING (lc) ORDER BY x; + +DROP TABLE nr; +DROP TABLE t; diff --git a/tests/queries/0_stateless/01227_distributed_global_in_issue_2610.sql b/tests/queries/0_stateless/01227_distributed_global_in_issue_2610.sql index a063e417e3a..781e9d6f8f9 100644 --- a/tests/queries/0_stateless/01227_distributed_global_in_issue_2610.sql +++ b/tests/queries/0_stateless/01227_distributed_global_in_issue_2610.sql @@ -4,3 +4,5 @@ create table data_01227 (key Int) Engine=MergeTree() order by key; insert into data_01227 select * from numbers(10); select * from remote('127.1', currentDatabase(), data_01227) prewhere key global in (select key from data_01227 prewhere key = 2); select * from cluster('test_cluster_two_shards', currentDatabase(), data_01227) prewhere key global in (select key from data_01227 prewhere key = 2); + +drop table data_01227; diff --git a/tests/queries/0_stateless/01231_distributed_aggregation_memory_efficient_mix_levels.sql b/tests/queries/0_stateless/01231_distributed_aggregation_memory_efficient_mix_levels.sql index 31f09b35bf3..80f7625900a 100644 --- a/tests/queries/0_stateless/01231_distributed_aggregation_memory_efficient_mix_levels.sql +++ b/tests/queries/0_stateless/01231_distributed_aggregation_memory_efficient_mix_levels.sql @@ -23,3 +23,8 @@ select x, count() from ma_dist group by x order by x; drop table if exists shard_0.shard_01231_distributed_aggregation_memory_efficient; drop table if exists shard_1.shard_01231_distributed_aggregation_memory_efficient; + +drop table ma_dist; + +drop database shard_0; +drop database shard_1; diff --git a/tests/queries/0_stateless/01231_operator_null_in.sql b/tests/queries/0_stateless/01231_operator_null_in.sql index ddebaf23900..27ab0bbd838 100644 --- a/tests/queries/0_stateless/01231_operator_null_in.sql +++ b/tests/queries/0_stateless/01231_operator_null_in.sql @@ -141,3 +141,4 @@ SELECT arraySort(x -> (x.1, x.2), groupArray(t)) == [(2, NULL), (NULL, NULL)] FR SELECT arraySort(x -> (x.1, x.2), groupArray(t)) == [] FROM null_in_tuple WHERE t not in ((1, '1'), (1, NULL), (2, NULL), (NULL, '3'), (NULL, NULL)); DROP TABLE IF EXISTS null_in_tuple; +DROP TABLE test_set2; diff --git a/tests/queries/0_stateless/01232_extremes.sql b/tests/queries/0_stateless/01232_extremes.sql index e46a6602766..b0670f731c6 100644 --- a/tests/queries/0_stateless/01232_extremes.sql +++ b/tests/queries/0_stateless/01232_extremes.sql @@ -52,3 +52,5 @@ drop table if exists shard_1.num2_01232; drop table if exists distr; drop table if exists distr2; +drop database shard_0; +drop database shard_1; diff --git a/tests/queries/0_stateless/01236_distributed_over_live_view_over_distributed.sql b/tests/queries/0_stateless/01236_distributed_over_live_view_over_distributed.sql index 618bdc1c5d2..522fc5dafbe 100644 --- a/tests/queries/0_stateless/01236_distributed_over_live_view_over_distributed.sql +++ b/tests/queries/0_stateless/01236_distributed_over_live_view_over_distributed.sql @@ -7,7 +7,7 @@ DROP TABLE IF EXISTS visits_layer; CREATE TABLE visits (StartDate Date) ENGINE MergeTree ORDER BY(StartDate); CREATE TABLE visits_layer (StartDate Date) ENGINE Distributed(test_cluster_two_shards_localhost, currentDatabase(), 'visits', rand()); -CREATE LIVE VIEW lv AS SELECT * FROM visits_layer ORDER BY StartDate; +CREATE LIVE VIEW lv AS SELECT * FROM visits_layer ORDER BY StartDate; CREATE TABLE visits_layer_lv (StartDate Date) ENGINE Distributed(test_cluster_two_shards_localhost, currentDatabase(), 'lv', rand()); @@ -20,3 +20,4 @@ DROP TABLE visits; DROP TABLE visits_layer; DROP TABLE lv; +DROP TABLE visits_layer_lv; diff --git a/tests/queries/0_stateless/01236_graphite_mt.sql b/tests/queries/0_stateless/01236_graphite_mt.sql index cee9b8c9fde..f3f1905b901 100644 --- a/tests/queries/0_stateless/01236_graphite_mt.sql +++ b/tests/queries/0_stateless/01236_graphite_mt.sql @@ -1,17 +1,17 @@ drop table if exists test_graphite; create table test_graphite (key UInt32, Path String, Time DateTime, Value Float64, Version UInt32, col UInt64) engine = GraphiteMergeTree('graphite_rollup') order by key settings index_granularity=10; -insert into test_graphite -select 1, 'sum_1', toDateTime(today()) - number * 60 - 30, number, 1, number from numbers(300) union all +insert into test_graphite +select 1, 'sum_1', toDateTime(today()) - number * 60 - 30, number, 1, number from numbers(300) union all select 2, 'sum_1', toDateTime(today()) - number * 60 - 30, number, 1, number from numbers(300) union all select 1, 'sum_2', toDateTime(today()) - number * 60 - 30, number, 1, number from numbers(300) union all select 2, 'sum_2', toDateTime(today()) - number * 60 - 30, number, 1, number from numbers(300) union all -select 1, 'max_1', toDateTime(today()) - number * 60 - 30, number, 1, number from numbers(300) union all +select 1, 'max_1', toDateTime(today()) - number * 60 - 30, number, 1, number from numbers(300) union all select 2, 'max_1', toDateTime(today()) - number * 60 - 30, number, 1, number from numbers(300) union all select 1, 'max_2', toDateTime(today()) - number * 60 - 30, number, 1, number from numbers(300) union all select 2, 'max_2', toDateTime(today()) - number * 60 - 30, number, 1, number from numbers(300); -insert into test_graphite +insert into test_graphite select 1, 'sum_1', toDateTime(today() - 3) - number * 60 - 30, number, 1, number from numbers(1200) union all select 2, 'sum_1', toDateTime(today() - 3) - number * 60 - 30, number, 1, number from numbers(1200) union all select 1, 'sum_2', toDateTime(today() - 3) - number * 60 - 30, number, 1, number from numbers(1200) union all @@ -24,3 +24,5 @@ select 2, 'max_2', toDateTime(today() - 3) - number * 60 - 30, number, 1, number optimize table test_graphite; select key, Path, Value, Version, col from test_graphite order by key, Path, Time desc; + +drop table test_graphite; diff --git a/tests/queries/0_stateless/01246_buffer_flush.sql b/tests/queries/0_stateless/01246_buffer_flush.sql index efe0adf703a..47891a7f00e 100644 --- a/tests/queries/0_stateless/01246_buffer_flush.sql +++ b/tests/queries/0_stateless/01246_buffer_flush.sql @@ -42,3 +42,5 @@ select 'drop'; insert into buffer_01256 select * from system.numbers limit 10; drop table if exists buffer_01256; select count() from data_01256; + +drop table data_01256; diff --git a/tests/queries/0_stateless/01246_finalize_aggregation_race.sql b/tests/queries/0_stateless/01246_finalize_aggregation_race.sql index 336fe6bcfea..c4946bd9e83 100644 --- a/tests/queries/0_stateless/01246_finalize_aggregation_race.sql +++ b/tests/queries/0_stateless/01246_finalize_aggregation_race.sql @@ -2,7 +2,7 @@ drop table if exists test_quantile; create table test_quantile (x AggregateFunction(quantileTiming(0.2), UInt64)) engine = Memory; insert into test_quantile select medianTimingState(.2)(number) from (select * from numbers(1000) order by number desc); select y from ( -select finalizeAggregation(x) as y from test_quantile union all +select finalizeAggregation(x) as y from test_quantile union all select finalizeAggregation(x) as y from test_quantile union all select finalizeAggregation(x) as y from test_quantile union all select finalizeAggregation(x) as y from test_quantile union all @@ -21,3 +21,4 @@ select finalizeAggregation(x) as y from test_quantile union all select finalizeAggregation(x) as y from test_quantile union all select finalizeAggregation(x) as y from test_quantile) order by y; +drop table test_quantile; diff --git a/tests/queries/0_stateless/01247_optimize_distributed_group_by_sharding_key_dist_on_dist.sql b/tests/queries/0_stateless/01247_optimize_distributed_group_by_sharding_key_dist_on_dist.sql index 7654ba71cc9..1000e956583 100644 --- a/tests/queries/0_stateless/01247_optimize_distributed_group_by_sharding_key_dist_on_dist.sql +++ b/tests/queries/0_stateless/01247_optimize_distributed_group_by_sharding_key_dist_on_dist.sql @@ -39,3 +39,4 @@ select count(), * from dist_01247 group by number order by number limit 1; drop table dist_01247; drop table dist_layer_01247; +drop table data_01247; diff --git a/tests/queries/0_stateless/01268_data_numeric_parameters.sql b/tests/queries/0_stateless/01268_data_numeric_parameters.sql index eceba51e7f5..3450fef9a6f 100644 --- a/tests/queries/0_stateless/01268_data_numeric_parameters.sql +++ b/tests/queries/0_stateless/01268_data_numeric_parameters.sql @@ -40,3 +40,7 @@ CREATE TABLE strings ( INSERT INTO strings VALUES ('test', 'string'); SELECT toTypeName(a), toTypeName(b) FROM strings; + +DROP TABLE floats; +DROP TABLE ints; +DROP TABLE strings; diff --git a/tests/queries/0_stateless/01268_mv_scalars.sql b/tests/queries/0_stateless/01268_mv_scalars.sql index fc082aaf88c..f6621502ca0 100644 --- a/tests/queries/0_stateless/01268_mv_scalars.sql +++ b/tests/queries/0_stateless/01268_mv_scalars.sql @@ -31,3 +31,5 @@ drop table dest_table_mv; drop table left_table; drop table right_table; drop table dest_table; +drop table src_table; +drop view dst_mv; diff --git a/tests/queries/0_stateless/01272_totals_and_filter_bug.sql b/tests/queries/0_stateless/01272_totals_and_filter_bug.sql index de751eb73bd..11992ab052a 100644 --- a/tests/queries/0_stateless/01272_totals_and_filter_bug.sql +++ b/tests/queries/0_stateless/01272_totals_and_filter_bug.sql @@ -9,22 +9,22 @@ INSERT INTO foo VALUES ('2020-01-01', 'test1', 10), ('2020-01-01', 'test2', 20); INSERT INTO bar VALUES ('2020-01-01', 'test2', 30), ('2020-01-01', 'test3', 40); SELECT - dimension_1, - sum_metric_1, + dimension_1, + sum_metric_1, sum_metric_2 -FROM +FROM ( SELECT - dimension_1, + dimension_1, sum(metric_1) AS sum_metric_1 FROM foo GROUP BY dimension_1 WITH TOTALS ) AS subquery_1 -ALL FULL OUTER JOIN +ALL FULL OUTER JOIN ( SELECT - dimension_1, + dimension_1, sum(metric_2) AS sum_metric_2 FROM bar GROUP BY dimension_1 @@ -32,3 +32,6 @@ ALL FULL OUTER JOIN ) AS subquery_2 USING (dimension_1) WHERE sum_metric_2 < 20 ORDER BY dimension_1 ASC; + +DROP TABLE foo; +DROP TABLE bar; diff --git a/tests/queries/0_stateless/01273_lc_fixed_string_field.sql b/tests/queries/0_stateless/01273_lc_fixed_string_field.sql index 18ef71fd6dc..11f93e918c6 100644 --- a/tests/queries/0_stateless/01273_lc_fixed_string_field.sql +++ b/tests/queries/0_stateless/01273_lc_fixed_string_field.sql @@ -13,3 +13,5 @@ INSERT INTO t (d, s, c) VALUES ('2020-01-01', 'ABC', 2); OPTIMIZE TABLE t; SELECT * FROM t; + +DROP TABLE t; diff --git a/tests/queries/0_stateless/01275_parallel_mv.sql b/tests/queries/0_stateless/01275_parallel_mv.sql index b67fbf02f8d..5d4dffae50a 100644 --- a/tests/queries/0_stateless/01275_parallel_mv.sql +++ b/tests/queries/0_stateless/01275_parallel_mv.sql @@ -16,3 +16,8 @@ select count() from testX; select count() from testXA; select count() from testXB; select count() from testXC; + +drop table testX; +drop view testXA; +drop view testXB; +drop view testXC; diff --git a/tests/queries/0_stateless/01279_dist_group_by.sql b/tests/queries/0_stateless/01279_dist_group_by.sql index 207b203f5d0..331efd4b687 100644 --- a/tests/queries/0_stateless/01279_dist_group_by.sql +++ b/tests/queries/0_stateless/01279_dist_group_by.sql @@ -7,3 +7,5 @@ set max_rows_to_group_by=10; set group_by_overflow_mode='any'; set group_by_two_level_threshold=100; select * from data_01279 group by key format Null; + +drop table data_01279; diff --git a/tests/queries/0_stateless/01283_max_threads_simple_query_optimization.sql b/tests/queries/0_stateless/01283_max_threads_simple_query_optimization.sql index 8de0f40229c..a7e51047eba 100644 --- a/tests/queries/0_stateless/01283_max_threads_simple_query_optimization.sql +++ b/tests/queries/0_stateless/01283_max_threads_simple_query_optimization.sql @@ -19,3 +19,5 @@ FROM system.query_log WHERE type = 'QueryFinish' AND query LIKE '%data_01283 LIMIT 1%' GROUP BY thread_ids FORMAT Null; + +DROP TABLE data_01283; diff --git a/tests/queries/0_stateless/001283_strict_resize_bug.reference b/tests/queries/0_stateless/01283_strict_resize_bug.reference similarity index 100% rename from tests/queries/0_stateless/001283_strict_resize_bug.reference rename to tests/queries/0_stateless/01283_strict_resize_bug.reference diff --git a/tests/queries/0_stateless/001283_strict_resize_bug.sql b/tests/queries/0_stateless/01283_strict_resize_bug.sql similarity index 90% rename from tests/queries/0_stateless/001283_strict_resize_bug.sql rename to tests/queries/0_stateless/01283_strict_resize_bug.sql index f462f50c61f..ee1aef05cd2 100644 --- a/tests/queries/0_stateless/001283_strict_resize_bug.sql +++ b/tests/queries/0_stateless/01283_strict_resize_bug.sql @@ -4,4 +4,4 @@ insert into num_10m select * from numbers(10000000); select * from (select sum(number) from num_10m union all select sum(number) from num_10m) limit 1 settings max_block_size = 1024; -drop table if exists num_1m; +drop table if exists num_10m; diff --git a/tests/queries/0_stateless/01285_data_skip_index_over_aggregation.sql b/tests/queries/0_stateless/01285_data_skip_index_over_aggregation.sql index 110c5b65cab..575b4dfa4a5 100644 --- a/tests/queries/0_stateless/01285_data_skip_index_over_aggregation.sql +++ b/tests/queries/0_stateless/01285_data_skip_index_over_aggregation.sql @@ -32,3 +32,5 @@ OPTIMIZE TABLE data_01285 FINAL; SELECT * FROM data_01285; -- and this passes even without fix. SELECT * FROM data_01285 WHERE assumeNotNull(value) = 3; + +DROP TABLE data_01285; diff --git a/tests/queries/0_stateless/01291_geo_types.sql b/tests/queries/0_stateless/01291_geo_types.sql index 0d923f08ccd..6b686ddf520 100644 --- a/tests/queries/0_stateless/01291_geo_types.sql +++ b/tests/queries/0_stateless/01291_geo_types.sql @@ -7,3 +7,5 @@ CREATE TABLE geo (a Point, b Ring, c Polygon, d MultiPolygon) ENGINE=Memory(); INSERT INTO geo VALUES((0, 0), [(0, 0), (10, 0), (10, 10), (0, 10)], [[(20, 20), (50, 20), (50, 50), (20, 50)], [(30, 30), (50, 50), (50, 30)]], [[[(0, 0), (10, 0), (10, 10), (0, 10)]], [[(20, 20), (50, 20), (50, 50), (20, 50)],[(30, 30), (50, 50), (50, 30)]]]); SELECT * from geo; + +DROP TABLE geo; diff --git a/tests/queries/0_stateless/01293_system_distribution_queue.sql b/tests/queries/0_stateless/01293_system_distribution_queue.sql index 4c9c690af09..8f84bbac41f 100644 --- a/tests/queries/0_stateless/01293_system_distribution_queue.sql +++ b/tests/queries/0_stateless/01293_system_distribution_queue.sql @@ -24,3 +24,6 @@ select is_blocked, error_count, data_files, data_compressed_bytes from system.di select 'UNBLOCK'; system start distributed sends dist_01293; select is_blocked, error_count, data_files, data_compressed_bytes from system.distribution_queue; + +drop table null_01293; +drop table dist_01293; diff --git a/tests/queries/0_stateless/01296_pipeline_stuck.sql b/tests/queries/0_stateless/01296_pipeline_stuck.sql index eeb67362634..2a23e6a9bf8 100644 --- a/tests/queries/0_stateless/01296_pipeline_stuck.sql +++ b/tests/queries/0_stateless/01296_pipeline_stuck.sql @@ -16,3 +16,5 @@ select 'INSERT SELECT max_insert_threads max_threads'; set max_insert_threads=2; insert into data_01295 select * from data_01295 final settings max_threads=2; -- no stuck for now select * from data_01295; + +drop table data_01295; diff --git a/tests/queries/0_stateless/01318_decrypt.sql b/tests/queries/0_stateless/01318_decrypt.sql index 796c42db1ab..fecca593272 100644 --- a/tests/queries/0_stateless/01318_decrypt.sql +++ b/tests/queries/0_stateless/01318_decrypt.sql @@ -150,3 +150,5 @@ WITH SELECT hex(decrypt('aes-256-gcm', concat(ciphertext, tag), key, iv, aad)) as plaintext_actual, plaintext_actual = hex(plaintext); + +DROP TABLE encryption_test; diff --git a/tests/queries/0_stateless/01318_encrypt.sql b/tests/queries/0_stateless/01318_encrypt.sql index 9766988764a..a9a60c89836 100644 --- a/tests/queries/0_stateless/01318_encrypt.sql +++ b/tests/queries/0_stateless/01318_encrypt.sql @@ -154,3 +154,5 @@ WITH SELECT hex(encrypt('aes-256-gcm', plaintext, key, iv, aad)) as ciphertext_actual, ciphertext_actual = concat(hex(ciphertext), hex(tag)); + +DROP TABLE encryption_test; diff --git a/tests/queries/0_stateless/01319_mv_constants_bug.sql b/tests/queries/0_stateless/01319_mv_constants_bug.sql index 191183ab286..4abb9d61b6c 100644 --- a/tests/queries/0_stateless/01319_mv_constants_bug.sql +++ b/tests/queries/0_stateless/01319_mv_constants_bug.sql @@ -22,3 +22,4 @@ DROP TABLE IF EXISTS distributed_table_1; DROP TABLE IF EXISTS distributed_table_2; DROP TABLE IF EXISTS local_table_1; DROP TABLE IF EXISTS local_table_2; +DROP TABLE local_table_merged; diff --git a/tests/queries/0_stateless/01319_optimize_skip_unused_shards_nesting.sql b/tests/queries/0_stateless/01319_optimize_skip_unused_shards_nesting.sql index b8a48c27e5f..4f7d6b81002 100644 --- a/tests/queries/0_stateless/01319_optimize_skip_unused_shards_nesting.sql +++ b/tests/queries/0_stateless/01319_optimize_skip_unused_shards_nesting.sql @@ -20,3 +20,7 @@ select * from dist_01319 where key = 1; set force_optimize_skip_unused_shards_nesting=2; set optimize_skip_unused_shards_nesting=1; select * from dist_01319 where key = 1; + +drop table data_01319; +drop table dist_01319; +drop table dist_layer_01319; diff --git a/tests/queries/0_stateless/01320_optimize_skip_unused_shards_no_non_deterministic.sql b/tests/queries/0_stateless/01320_optimize_skip_unused_shards_no_non_deterministic.sql index ca58f7be94c..f827bb733b4 100644 --- a/tests/queries/0_stateless/01320_optimize_skip_unused_shards_no_non_deterministic.sql +++ b/tests/queries/0_stateless/01320_optimize_skip_unused_shards_no_non_deterministic.sql @@ -8,3 +8,6 @@ create table dist_01320 as data_01320 Engine=Distributed(test_cluster_two_shards set optimize_skip_unused_shards=1; set force_optimize_skip_unused_shards=1; select * from dist_01320 where key = 0; -- { serverError 507 } + +drop table data_01320; +drop table dist_01320; diff --git a/tests/queries/0_stateless/01346_alter_enum_partition_key_replicated_zookeeper.sql b/tests/queries/0_stateless/01346_alter_enum_partition_key_replicated_zookeeper.sql index 0719282734a..d185973f564 100644 --- a/tests/queries/0_stateless/01346_alter_enum_partition_key_replicated_zookeeper.sql +++ b/tests/queries/0_stateless/01346_alter_enum_partition_key_replicated_zookeeper.sql @@ -47,3 +47,4 @@ ALTER TABLE test DROP COLUMN x; -- { serverError 47 } ALTER TABLE test DROP COLUMN y; -- { serverError 47 } DROP TABLE test; +DROP TABLE test2; diff --git a/tests/queries/0_stateless/01392_column_resolve.sql b/tests/queries/0_stateless/01392_column_resolve.sql index 93d84359e1f..728ccfd0728 100644 --- a/tests/queries/0_stateless/01392_column_resolve.sql +++ b/tests/queries/0_stateless/01392_column_resolve.sql @@ -1,15 +1,14 @@ -DROP TABLE IF EXISTS tableConversion; -DROP TABLE IF EXISTS tableClick; -DROP TABLE IF EXISTS leftjoin; +DROP DATABASE IF EXISTS test_01392; +CREATE DATABASE test_01392; -CREATE TABLE default.tableConversion (conversionId String, value Nullable(Double)) ENGINE = Log(); -CREATE TABLE default.tableClick (clickId String, conversionId String, value Nullable(Double)) ENGINE = Log(); -CREATE TABLE default.leftjoin (id String) ENGINE = Log(); +CREATE TABLE test_01392.tableConversion (conversionId String, value Nullable(Double)) ENGINE = Log(); +CREATE TABLE test_01392.tableClick (clickId String, conversionId String, value Nullable(Double)) ENGINE = Log(); +CREATE TABLE test_01392.leftjoin (id String) ENGINE = Log(); -INSERT INTO default.tableConversion(conversionId, value) VALUES ('Conversion 1', 1); -INSERT INTO default.tableClick(clickId, conversionId, value) VALUES ('Click 1', 'Conversion 1', 14); -INSERT INTO default.tableClick(clickId, conversionId, value) VALUES ('Click 2', 'Conversion 1', 15); -INSERT INTO default.tableClick(clickId, conversionId, value) VALUES ('Click 3', 'Conversion 1', 16); +INSERT INTO test_01392.tableConversion(conversionId, value) VALUES ('Conversion 1', 1); +INSERT INTO test_01392.tableClick(clickId, conversionId, value) VALUES ('Click 1', 'Conversion 1', 14); +INSERT INTO test_01392.tableClick(clickId, conversionId, value) VALUES ('Click 2', 'Conversion 1', 15); +INSERT INTO test_01392.tableClick(clickId, conversionId, value) VALUES ('Click 3', 'Conversion 1', 16); SELECT conversion.conversionId AS myConversionId, @@ -17,17 +16,19 @@ SELECT click.myValue AS myValue FROM ( SELECT conversionId, value as myValue - FROM default.tableConversion + FROM test_01392.tableConversion ) AS conversion INNER JOIN ( SELECT clickId, conversionId, value as myValue - FROM default.tableClick + FROM test_01392.tableClick ) AS click ON click.conversionId = conversion.conversionId LEFT JOIN ( - SELECT * FROM default.leftjoin + SELECT * FROM test_01392.leftjoin ) AS dummy ON (dummy.id = conversion.conversionId) ORDER BY myValue; -DROP TABLE IF EXISTS tableConversion; -DROP TABLE IF EXISTS tableClick; -DROP TABLE IF EXISTS leftjoin; +DROP TABLE test_01392.tableConversion; +DROP TABLE test_01392.tableClick; +DROP TABLE test_01392.leftjoin; + +DROP DATABASE test_01392; diff --git a/tests/queries/0_stateless/01400_join_get_with_multi_keys.sql b/tests/queries/0_stateless/01400_join_get_with_multi_keys.sql index 8a19865359b..8f83e1c15dd 100644 --- a/tests/queries/0_stateless/01400_join_get_with_multi_keys.sql +++ b/tests/queries/0_stateless/01400_join_get_with_multi_keys.sql @@ -13,3 +13,4 @@ INSERT INTO test_lc VALUES ('ab', '1', 0.1), ('ab', '2', 0.2), ('cd', '3', 0.3); SELECT joinGet(test_lc, 'c', 'ab', '1'); DROP TABLE test_joinGet; +DROP TABLE test_lc; diff --git a/tests/queries/0_stateless/01409_topK_merge.sql b/tests/queries/0_stateless/01409_topK_merge.sql index 5ac7c350093..d7659351540 100644 --- a/tests/queries/0_stateless/01409_topK_merge.sql +++ b/tests/queries/0_stateless/01409_topK_merge.sql @@ -11,3 +11,5 @@ select length(topKWeighted(20)(number, 1)) from remote('127.{1,1}', currentDatab select 'AggregateFunctionTopKGenericData'; select length(topK(20)((number, ''))) from remote('127.{1,1}', currentDatabase(), data_01409); select length(topKWeighted(20)((number, ''), 1)) from remote('127.{1,1}', currentDatabase(), data_01409); + +drop table data_01409; diff --git a/tests/queries/0_stateless/01413_truncate_without_table_keyword.sql b/tests/queries/0_stateless/01413_truncate_without_table_keyword.sql index 266c5b84be3..c6819f77e80 100644 --- a/tests/queries/0_stateless/01413_truncate_without_table_keyword.sql +++ b/tests/queries/0_stateless/01413_truncate_without_table_keyword.sql @@ -10,3 +10,4 @@ TRUNCATE truncate_test; SELECT * FROM truncate_test ORDER BY uint8; +DROP TABLE truncate_test; diff --git a/tests/queries/0_stateless/01419_skip_index_compact_parts.sql b/tests/queries/0_stateless/01419_skip_index_compact_parts.sql index 8b03f28df44..580fcf7f41e 100644 --- a/tests/queries/0_stateless/01419_skip_index_compact_parts.sql +++ b/tests/queries/0_stateless/01419_skip_index_compact_parts.sql @@ -1,5 +1,5 @@ DROP TABLE IF EXISTS index_compact; - + CREATE TABLE index_compact(a UInt32, b UInt32, index i1 b type minmax granularity 1) ENGINE = MergeTree ORDER BY a SETTINGS min_rows_for_wide_part = 1000, index_granularity = 128, merge_max_block_size = 100; @@ -10,3 +10,5 @@ INSERT INTO index_compact SELECT number, toString(number) FROM numbers(30); OPTIMIZE TABLE index_compact FINAL; SELECT count() FROM index_compact WHERE b < 10; + +DROP TABLE index_compact; diff --git a/tests/queries/0_stateless/01456_low_cardinality_sorting_bugfix.sql b/tests/queries/0_stateless/01456_low_cardinality_sorting_bugfix.sql index 507a798e7b6..06412d4b062 100644 --- a/tests/queries/0_stateless/01456_low_cardinality_sorting_bugfix.sql +++ b/tests/queries/0_stateless/01456_low_cardinality_sorting_bugfix.sql @@ -39,3 +39,5 @@ SELECT cast(color,'String') color, timestamp FROM order_test1 GROUP BY color, timestamp ORDER BY color ASC, timestamp DESC; + +DROP TABLE order_test1; diff --git a/tests/queries/0_stateless/01457_create_as_table_function_structure.sql b/tests/queries/0_stateless/01457_create_as_table_function_structure.sql index 1bb0d9c54ca..1c9c1e1ef44 100644 --- a/tests/queries/0_stateless/01457_create_as_table_function_structure.sql +++ b/tests/queries/0_stateless/01457_create_as_table_function_structure.sql @@ -32,3 +32,4 @@ SELECT (*,).1 AS c, toTypeName(c) FROM test_01457.tf_merge; DROP DATABASE test_01457; +DROP TABLE tmp; diff --git a/tests/queries/0_stateless/01460_DistributedFilesToInsert.sql b/tests/queries/0_stateless/01460_DistributedFilesToInsert.sql index 8ce10cf133f..34c0d55d573 100644 --- a/tests/queries/0_stateless/01460_DistributedFilesToInsert.sql +++ b/tests/queries/0_stateless/01460_DistributedFilesToInsert.sql @@ -40,3 +40,5 @@ select sleep(1) format Null; -- distributed_directory_monitor_sleep_time_ms select value from system.metrics where metric = 'DistributedFilesToInsert'; drop table dist_01460; select value from system.metrics where metric = 'DistributedFilesToInsert'; + +drop table data_01460; diff --git a/tests/queries/0_stateless/01463_test_alter_live_view_refresh.sql b/tests/queries/0_stateless/01463_test_alter_live_view_refresh.sql index ab316a377fd..04276087ece 100644 --- a/tests/queries/0_stateless/01463_test_alter_live_view_refresh.sql +++ b/tests/queries/0_stateless/01463_test_alter_live_view_refresh.sql @@ -8,3 +8,6 @@ CREATE LIVE VIEW live1 AS SELECT * FROM test0; select 'ALTER LIVE VIEW live1 REFRESH'; ALTER LIVE VIEW live1 REFRESH; -- success + +DROP TABLE test0; +DROP VIEW live1; diff --git a/tests/queries/0_stateless/01487_distributed_in_not_default_db.sql b/tests/queries/0_stateless/01487_distributed_in_not_default_db.sql index f6f7471711a..39e04804bfb 100644 --- a/tests/queries/0_stateless/01487_distributed_in_not_default_db.sql +++ b/tests/queries/0_stateless/01487_distributed_in_not_default_db.sql @@ -34,3 +34,4 @@ DROP TABLE IF EXISTS t; DROP DATABASE shard_0; DROP DATABASE shard_1; +DROP DATABASE main_01487; diff --git a/tests/queries/0_stateless/01511_prewhere_with_virtuals.sql b/tests/queries/0_stateless/01511_prewhere_with_virtuals.sql index 8b272c8cff1..43f003fc80f 100644 --- a/tests/queries/0_stateless/01511_prewhere_with_virtuals.sql +++ b/tests/queries/0_stateless/01511_prewhere_with_virtuals.sql @@ -1,6 +1,6 @@ DROP TABLE IF EXISTS test_not_found_column_nothing; -CREATE TABLE test_not_found_column_nothing +CREATE TABLE test_not_found_column_nothing ( col001 UInt8, col002 UInt8 @@ -10,3 +10,5 @@ INSERT INTO test_not_found_column_nothing(col001) SELECT number FROM numbers(11) SELECT _part, count() FROM test_not_found_column_nothing PREWHERE col001 % 3 != 0 GROUP BY _part ORDER BY _part; SELECT _part FROM test_not_found_column_nothing PREWHERE col001 = 0; + +DROP TABLE test_not_found_column_nothing; diff --git a/tests/queries/0_stateless/01513_optimize_aggregation_in_order_memory.sql b/tests/queries/0_stateless/01513_optimize_aggregation_in_order_memory.sql index 38920262fba..6aa38a914f7 100644 --- a/tests/queries/0_stateless/01513_optimize_aggregation_in_order_memory.sql +++ b/tests/queries/0_stateless/01513_optimize_aggregation_in_order_memory.sql @@ -14,3 +14,5 @@ select key, groupArray(repeat('a', 200)), count() from data_01513 group by key f select key, groupArray(repeat('a', 200)), count() from data_01513 group by key format Null settings optimize_aggregation_in_order=1; -- for WITH TOTALS previous groups should be kept. select key, groupArray(repeat('a', 200)), count() from data_01513 group by key with totals format Null settings optimize_aggregation_in_order=1; -- { serverError 241; } + +drop table data_01513; diff --git a/tests/queries/0_stateless/01514_empty_buffer_different_types.sql b/tests/queries/0_stateless/01514_empty_buffer_different_types.sql index 3afadbcd33f..a1debbf7e13 100644 --- a/tests/queries/0_stateless/01514_empty_buffer_different_types.sql +++ b/tests/queries/0_stateless/01514_empty_buffer_different_types.sql @@ -10,3 +10,4 @@ select s from buffer_table1 where x = 1; select s from buffer_table1 where x = 2; DROP TABLE IF EXISTS merge_tree_table1; +DROP TABLE buffer_table1; diff --git a/tests/queries/0_stateless/01515_force_data_skipping_indices.sql b/tests/queries/0_stateless/01515_force_data_skipping_indices.sql index 5e45018707d..53d3e5c736f 100644 --- a/tests/queries/0_stateless/01515_force_data_skipping_indices.sql +++ b/tests/queries/0_stateless/01515_force_data_skipping_indices.sql @@ -29,3 +29,5 @@ SELECT * FROM data_01515 WHERE d1 = 0 SETTINGS force_data_skipping_indices=' d1 SELECT * FROM data_01515 WHERE d1_null = 0 SETTINGS force_data_skipping_indices='d1_null_idx'; -- { serverError 277 } SELECT * FROM data_01515 WHERE assumeNotNull(d1_null) = 0 SETTINGS force_data_skipping_indices='d1_null_idx'; + +DROP TABLE data_01515; diff --git a/tests/queries/0_stateless/01515_mv_and_array_join_optimisation_bag.sql b/tests/queries/0_stateless/01515_mv_and_array_join_optimisation_bag.sql index 9cca5e85da7..ad762ea65fb 100644 --- a/tests/queries/0_stateless/01515_mv_and_array_join_optimisation_bag.sql +++ b/tests/queries/0_stateless/01515_mv_and_array_join_optimisation_bag.sql @@ -24,14 +24,14 @@ SELECT sumIfState(Sign, _uniq = 1) AS Visits, sumState(Sign) AS GoalReaches FROM visits -ARRAY JOIN +ARRAY JOIN GoalsID AS GoalID, arrayEnumerateUniq(GoalsID) AS _uniq -GROUP BY +GROUP BY CounterID, StartDate, GoalID -ORDER BY +ORDER BY CounterID ASC, StartDate ASC, GoalID ASC; @@ -46,3 +46,7 @@ CREATE TABLE goal ) ENGINE = AggregatingMergeTree PARTITION BY toStartOfMonth(StartDate) ORDER BY (CounterID, StartDate, GoalID) SETTINGS index_granularity = 256; INSERT INTO visits (`CounterID`,`StartDate`,`StartTime`,`Sign`,`GoalsID`) VALUES (1, toDate('2000-01-01'), toDateTime(toDate('2000-01-01')), 1, [1]); + +DROP TABLE goal; +DROP TABLE goal_view; +DROP TABLE visits; diff --git a/tests/queries/0_stateless/01517_drop_mv_with_inner_table.sql b/tests/queries/0_stateless/01517_drop_mv_with_inner_table.sql index bbf7f0d04be..b97480c2911 100644 --- a/tests/queries/0_stateless/01517_drop_mv_with_inner_table.sql +++ b/tests/queries/0_stateless/01517_drop_mv_with_inner_table.sql @@ -38,3 +38,10 @@ create materialized view db_01517_ordinary.mv engine=Null as select * from db_01 -- drops it and hangs with Atomic engine, due to recursive DROP drop table db_01517_ordinary.mv sync; show tables from db_01517_ordinary; + +drop table db_01517_atomic_sync.source; +drop table db_01517_ordinary.source; + +drop database db_01517_atomic; +drop database db_01517_atomic_sync; +drop database db_01517_ordinary; diff --git a/tests/queries/0_stateless/01526_initial_query_id.sh b/tests/queries/0_stateless/01526_initial_query_id.sh index c5459625023..e28f9ee1e40 100755 --- a/tests/queries/0_stateless/01526_initial_query_id.sh +++ b/tests/queries/0_stateless/01526_initial_query_id.sh @@ -10,7 +10,7 @@ ${CLICKHOUSE_CLIENT} -q "select 1 format Null" "--query_id=$query_id" ${CLICKHOUSE_CURL} \ --header "X-ClickHouse-Query-Id: $query_id" \ - "http://localhost:8123/" \ + $CLICKHOUSE_URL \ --get \ --data-urlencode "query=select 1 format Null" diff --git a/tests/queries/0_stateless/01528_allow_nondeterministic_optimize_skip_unused_shards.sql b/tests/queries/0_stateless/01528_allow_nondeterministic_optimize_skip_unused_shards.sql index b0bfb2aae3f..51970db73cb 100644 --- a/tests/queries/0_stateless/01528_allow_nondeterministic_optimize_skip_unused_shards.sql +++ b/tests/queries/0_stateless/01528_allow_nondeterministic_optimize_skip_unused_shards.sql @@ -5,3 +5,5 @@ set optimize_skip_unused_shards=1; set force_optimize_skip_unused_shards=1; select * from dist_01528 where dummy = 2; -- { serverError 507; } select * from dist_01528 where dummy = 2 settings allow_nondeterministic_optimize_skip_unused_shards=1; + +drop table dist_01528; diff --git a/tests/queries/0_stateless/01530_drop_database_atomic_sync.sql b/tests/queries/0_stateless/01530_drop_database_atomic_sync.sql index 010b8931448..d5fb25a9241 100644 --- a/tests/queries/0_stateless/01530_drop_database_atomic_sync.sql +++ b/tests/queries/0_stateless/01530_drop_database_atomic_sync.sql @@ -28,7 +28,7 @@ drop database db_01530_atomic; create database db_01530_atomic Engine=Atomic; create table db_01530_atomic.data (key Int) Engine=ReplicatedMergeTree('/clickhouse/tables/db_01530_atomic/data', 'test') order by key; -- { serverError 253; } --- TODO: SYSTEM FORCE DROP and uncomment the line below to cleanup the data after test --- (otherwise the test is not retriable...) --- --- drop database db_01530_atomic sync; + +set database_atomic_wait_for_drop_and_detach_synchronously=1; + +drop database db_01530_atomic sync; diff --git a/tests/queries/0_stateless/01551_mergetree_read_in_order_spread.sql b/tests/queries/0_stateless/01551_mergetree_read_in_order_spread.sql index 831a7282861..a71c9f9a714 100644 --- a/tests/queries/0_stateless/01551_mergetree_read_in_order_spread.sql +++ b/tests/queries/0_stateless/01551_mergetree_read_in_order_spread.sql @@ -14,3 +14,5 @@ SET merge_tree_min_rows_for_concurrent_read=10000; SET optimize_aggregation_in_order=1; SET read_in_order_two_level_merge_threshold=1; EXPLAIN PIPELINE SELECT key FROM data_01551 GROUP BY key, key/2; + +DROP TABLE data_01551; diff --git a/tests/queries/0_stateless/arcadia_skip_list.txt b/tests/queries/0_stateless/arcadia_skip_list.txt index 66812f6e16b..711a69e0282 100644 --- a/tests/queries/0_stateless/arcadia_skip_list.txt +++ b/tests/queries/0_stateless/arcadia_skip_list.txt @@ -166,4 +166,5 @@ 01548_query_log_query_execution_ms 01552_dict_fixedstring 01555_system_distribution_queue_mask -01557_max_parallel_replicas_no_sample.sql \ No newline at end of file +01557_max_parallel_replicas_no_sample.sql +01525_select_with_offset_fetch_clause diff --git a/tests/queries/conftest.py b/tests/queries/conftest.py index 85b7250af0c..2f19ae7c479 100644 --- a/tests/queries/conftest.py +++ b/tests/queries/conftest.py @@ -36,6 +36,11 @@ def sql_query(request): return os.path.join(QUERIES_PATH, os.path.splitext(request.param)[0]) +@pytest.fixture(scope='module', params=[f for f in os.listdir(QUERIES_PATH) if f.endswith('.sh')]) +def shell_query(request): + return os.path.join(QUERIES_PATH, os.path.splitext(request.param)[0]) + + @pytest.fixture def standalone_server(bin_prefix, tmp_path): server = ServerThread(bin_prefix, str(tmp_path)) diff --git a/tests/queries/query_test.py b/tests/queries/query_test.py index d0fc5759667..adaf9bc4e5e 100644 --- a/tests/queries/query_test.py +++ b/tests/queries/query_test.py @@ -8,18 +8,16 @@ import subprocess import sys -def run_client(bin_prefix, port, query, reference, replace_map={}): - client = subprocess.Popen([bin_prefix + '-client', '--port', str(port), '-m', '-n', '--testmode'], - stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE) - result, error = client.communicate(query.encode('utf-8')) - assert client.returncode is not None, "Client should exit after processing all queries" - +def check_result(result, error, return_code, reference, replace_map): for old, new in replace_map.items(): result = result.replace(old.encode('utf-8'), new.encode('utf-8')) - if client.returncode != 0: - print(error.decode('utf-8'), file=sys.stderr) - pytest.fail('Client died unexpectedly with code {code}'.format(code=client.returncode), pytrace=False) + if return_code != 0: + try: + print(error.decode('utf-8'), file=sys.stderr) + except UnicodeDecodeError: + print(error.decode('latin1'), file=sys.stderr) # encoding with 1 symbol per 1 byte, covering all values + pytest.fail('Client died unexpectedly with code {code}'.format(code=return_code), pytrace=False) elif result != reference: pytest.fail("Query output doesn't match reference:{eol}{diff}".format( eol=os.linesep, @@ -29,13 +27,38 @@ def run_client(bin_prefix, port, query, reference, replace_map={}): pytrace=False) +def run_client(bin_prefix, port, query, reference, replace_map={}): + # We can't use `text=True` since some tests may return binary data + client = subprocess.Popen([bin_prefix + '-client', '--port', str(port), '-m', '-n', '--testmode'], + stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + result, error = client.communicate(query.encode('utf-8')) + assert client.returncode is not None, "Client should exit after processing all queries" + + check_result(result, error, client.returncode, reference, replace_map) + + +def run_shell(bin_prefix, tmp_dir, tcp_port, http_port, inter_port, database, path, reference, replace_map={}): + env = { + 'CLICKHOUSE_BINARY': bin_prefix, + 'CLICKHOUSE_DATABASE': database, + 'CLICKHOUSE_PORT_TCP': str(tcp_port), + 'CLICKHOUSE_PORT_HTTP': str(http_port), + 'CLICKHOUSE_PORT_INTERSERVER': str(inter_port), + 'CLICKHOUSE_TMP': tmp_dir, + } + shell = subprocess.Popen([path], env=env, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + result, error = shell.communicate() + assert shell.returncode is not None, "Script should exit after executing all commands" + + check_result(result, error, shell.returncode, reference, replace_map) + + def random_str(length=10): alphabet = string.ascii_lowercase + string.digits return ''.join(random.choice(alphabet) for _ in range(length)) -@pytest.mark.timeout(timeout=30, method='signal') -def test_query(bin_prefix, sql_query, standalone_server): +def test_sql_query(bin_prefix, sql_query, standalone_server): tcp_port = standalone_server.tcp_port query_path = sql_query + ".sql" @@ -56,7 +79,39 @@ def test_query(bin_prefix, sql_query, standalone_server): query = "SELECT 'SHOW ORPHANED TABLES'; SELECT name FROM system.tables WHERE database != 'system' ORDER BY (database, name);" run_client(bin_prefix, tcp_port, query, b'SHOW ORPHANED TABLES\n') - run_client(bin_prefix, tcp_port, 'DROP DATABASE {random};'.format(random=random_name), b'') + query = 'DROP DATABASE {random};'.format(random=random_name) + run_client(bin_prefix, tcp_port, query, b'') + + query = "SELECT 'SHOW ORPHANED DATABASES'; SHOW DATABASES;" + run_client(bin_prefix, tcp_port, query, b'SHOW ORPHANED DATABASES\n_temporary_and_external_tables\ndefault\nsystem\n') + + +def test_shell_query(bin_prefix, shell_query, standalone_server): + tcp_port = standalone_server.tcp_port + http_port = standalone_server.http_port + inter_port = standalone_server.inter_port + tmp_path = standalone_server.tmp_dir + + shell_path = shell_query + ".sh" + reference_path = shell_query + ".reference" + + if not os.path.exists(reference_path): + pytest.skip('No .reference file found') + + with open(reference_path, 'rb') as file: + reference = file.read() + + random_name = 'test_{random}'.format(random=random_str()) + query = 'CREATE DATABASE {random};'.format(random=random_name) + run_client(bin_prefix, tcp_port, query, b'') + + run_shell(bin_prefix, tmp_path, tcp_port, http_port, inter_port, random_name, shell_path, reference, {random_name: 'default'}) + + query = "SELECT 'SHOW ORPHANED TABLES'; SELECT name FROM system.tables WHERE database != 'system' ORDER BY (database, name);" + run_client(bin_prefix, tcp_port, query, b'SHOW ORPHANED TABLES\n') + + query = 'DROP DATABASE {random};'.format(random=random_name) + run_client(bin_prefix, tcp_port, query, b'') query = "SELECT 'SHOW ORPHANED DATABASES'; SHOW DATABASES;" run_client(bin_prefix, tcp_port, query, b'SHOW ORPHANED DATABASES\n_temporary_and_external_tables\ndefault\nsystem\n') diff --git a/tests/queries/server.py b/tests/queries/server.py index c4f8968e08a..b3c3ece0b4c 100644 --- a/tests/queries/server.py +++ b/tests/queries/server.py @@ -23,6 +23,7 @@ class ServerThread(threading.Thread): self.etc_dir = os.path.join(tmp_dir, 'etc') self.server_config = os.path.join(self.etc_dir, 'server-config.xml') self.users_config = os.path.join(self.etc_dir, 'users.xml') + self.dicts_config = os.path.join(self.etc_dir, 'dictionaries.xml') os.makedirs(self.log_dir) os.makedirs(self.etc_dir) @@ -51,6 +52,9 @@ class ServerThread(threading.Thread): with open(self.users_config, 'w') as f: f.write(ServerThread.DEFAULT_USERS_CONFIG) + with open(self.dicts_config, 'w') as f: + f.write(ServerThread.DEFAULT_DICTIONARIES_CONFIG.format(tcp_port=self.tcp_port)) + def run(self): retries = ServerThread.DEFAULT_RETRIES @@ -122,10 +126,45 @@ ServerThread.DEFAULT_SERVER_CONFIG = \ {tmp_dir}/tmp/ {tmp_dir}/data/access/ users.xml + dictionaries.xml 5368709120 + 3 Europe/Moscow + + Hello, world! + s1 + r1 + + + + Version + + sum + sum + + 0 + 600 + + + 172800 + 6000 + + + + max + + 0 + 600 + + + 172800 + 6000 + + + + @@ -180,7 +219,78 @@ ServerThread.DEFAULT_SERVER_CONFIG = \ + + + + + shard_0 + localhost + {tcp_port} + + + + + shard_1 + localhost + {tcp_port} + + + + + + + true + + 127.0.0.1 + {tcp_port} + + + + true + + 127.0.0.2 + {tcp_port} + + + + + + + + memory + + + + + + testkeeper + + + + system + part_log
+ + + + system + query_log
+ + + + system + query_thread_log
+ + + + system + text_log
+ + + + system + trace_log
+ """ @@ -242,3 +352,735 @@ ServerThread.DEFAULT_USERS_CONFIG = \ """ + + +ServerThread.DEFAULT_DICTIONARIES_CONFIG = \ +"""\ + + + flat_ints + + + localhost + {tcp_port} + default + + system + ints
+ + + 0 + + + + + + key + + + i8 + Int8 + 0 + + + i16 + Int16 + 0 + + + i32 + Int32 + 0 + + + i64 + Int64 + 0 + + + u8 + UInt8 + 0 + + + u16 + UInt16 + 0 + + + u32 + UInt32 + 0 + + + u64 + UInt64 + 0 + + + + + + hashed_ints + + + localhost + {tcp_port} + default + + system + ints
+ + + 0 + + + + + + key + + + i8 + Int8 + 0 + + + i16 + Int16 + 0 + + + i32 + Int32 + 0 + + + i64 + Int64 + 0 + + + u8 + UInt8 + 0 + + + u16 + UInt16 + 0 + + + u32 + UInt32 + 0 + + + u64 + UInt64 + 0 + + + + + + hashed_sparse_ints + + + localhost + {tcp_port} + default + + system + ints
+ + + 0 + + + + + + key + + + i8 + Int8 + 0 + + + i16 + Int16 + 0 + + + i32 + Int32 + 0 + + + i64 + Int64 + 0 + + + u8 + UInt8 + 0 + + + u16 + UInt16 + 0 + + + u32 + UInt32 + 0 + + + u64 + UInt64 + 0 + + + + + + cache_ints + + + localhost + {tcp_port} + default + + system + ints
+ + + 0 + + 1000 + + + + key + + + i8 + Int8 + 0 + + + i16 + Int16 + 0 + + + i32 + Int32 + 0 + + + i64 + Int64 + 0 + + + u8 + UInt8 + 0 + + + u16 + UInt16 + 0 + + + u32 + UInt32 + 0 + + + u64 + UInt64 + 0 + + + + + + complex_hashed_ints + + + localhost + {tcp_port} + default + + system + ints
+ + + 0 + + + + + + + key + UInt64 + + + + i8 + Int8 + 0 + + + i16 + Int16 + 0 + + + i32 + Int32 + 0 + + + i64 + Int64 + 0 + + + u8 + UInt8 + 0 + + + u16 + UInt16 + 0 + + + u32 + UInt32 + 0 + + + u64 + UInt64 + 0 + + + + + + complex_cache_ints + + + localhost + {tcp_port} + default + + system + ints
+ + + 0 + + 1000 + + + + + key + UInt64 + + + + i8 + Int8 + 0 + + + i16 + Int16 + 0 + + + i32 + Int32 + 0 + + + i64 + Int64 + 0 + + + u8 + UInt8 + 0 + + + u16 + UInt16 + 0 + + + u32 + UInt32 + 0 + + + u64 + UInt64 + 0 + + + + + + flat_strings + + + localhost + {tcp_port} + default + + system + strings
+ + + 0 + + + + + + key + + + str + String + + + + + + + hashed_strings + + + localhost + {tcp_port} + default + + system + strings
+ + + 0 + + + + + + key + + + str + String + + + + + + + cache_strings + + + localhost + {tcp_port} + default + + system + strings
+ + + 0 + + 1000 + + + + key + + + str + String + + + + + + + complex_hashed_strings + + + localhost + {tcp_port} + default + + system + strings
+ + + 0 + + + + + + + key + UInt64 + + + + str + String + + + + + + + complex_cache_strings + + + localhost + {tcp_port} + default + + system + strings
+ + + 0 + + 1000 + + + + + key + UInt64 + + + + str + String + + + + + + + flat_decimals + + + localhost + {tcp_port} + default + + system + decimals
+ + + 0 + + + + + + key + + + d32 + Decimal32(4) + 0 + + + d64 + Decimal64(6) + 0 + + + d128 + Decimal128(1) + 0 + + + + + + hashed_decimals + + + localhost + {tcp_port} + default + + system + decimals
+ + + 0 + + + + + + key + + + d32 + Decimal32(4) + 0 + + + d64 + Decimal64(6) + 0 + + + d128 + Decimal128(1) + 0 + + + + + + cache_decimals + + + localhost + {tcp_port} + default + + system + decimals
+ + + 0 + + 1000 + + + + key + + + d32 + Decimal32(4) + 0 + + + d64 + Decimal64(6) + 0 + + + d128 + Decimal128(1) + 0 + + + + + + complex_hashed_decimals + + + localhost + {tcp_port} + default + + system + decimals
+ + + 0 + + + + + + + key + UInt64 + + + + d32 + Decimal32(4) + 0 + + + d64 + Decimal64(6) + 0 + + + d128 + Decimal128(1) + 0 + + + + + + complex_cache_decimals + + + localhost + {tcp_port} + default + + system + decimals
+ + + 0 + + 1000 + + + + + key + UInt64 + + + + d32 + Decimal32(4) + 0 + + + d64 + Decimal64(6) + 0 + + + d128 + Decimal128(1) + 0 + + + + +""" diff --git a/tests/queries/shell_config.sh b/tests/queries/shell_config.sh index 0b5b0940cd7..9cfec2c2a28 100644 --- a/tests/queries/shell_config.sh +++ b/tests/queries/shell_config.sh @@ -1,5 +1,8 @@ #!/usr/bin/env bash +# Don't check for ODR violation, since we may test shared build with ASAN +export ASAN_OPTIONS=detect_odr_violation=0 + export CLICKHOUSE_DATABASE=${CLICKHOUSE_DATABASE:="test"} export CLICKHOUSE_CLIENT_SERVER_LOGS_LEVEL=${CLICKHOUSE_CLIENT_SERVER_LOGS_LEVEL:="warning"} [ -v CLICKHOUSE_CONFIG_CLIENT ] && CLICKHOUSE_CLIENT_OPT0+=" --config-file=${CLICKHOUSE_CONFIG_CLIENT} " diff --git a/utils/check-mysql-binlog/main.cpp b/utils/check-mysql-binlog/main.cpp index 4ec40ac41cc..0d831b84dce 100644 --- a/utils/check-mysql-binlog/main.cpp +++ b/utils/check-mysql-binlog/main.cpp @@ -9,6 +9,7 @@ #include #include #include +#include #include static DB::MySQLReplication::BinlogEventPtr parseSingleEventBody( @@ -126,18 +127,20 @@ static int checkBinLogFile(const std::string & bin_path, bool exist_checksum) } catch (...) { - std::cerr << "Unable to parse MySQL binlog event. Code: " << DB::getCurrentExceptionCode() << ", Exception message: " - << DB::getCurrentExceptionMessage(false) << std::endl << ", Previous event: " << std::endl; - last_event->dump(std::cerr); - std::cerr << std::endl << ", Event header: " << std::endl; - last_header->dump(std::cerr); - std::cerr << std::endl; + DB::WriteBufferFromOStream cerr(std::cerr); + cerr << "Unable to parse MySQL binlog event. Code: " << DB::getCurrentExceptionCode() << ", Exception message: " + << DB::getCurrentExceptionMessage(false) << '\n' << ", Previous event: " << '\n'; + last_event->dump(cerr); + cerr << '\n' << ", Event header: " << '\n'; + last_header->dump(cerr); + cerr << '\n'; return DB::getCurrentExceptionCode(); } - std::cout << "Check passed. " << std::endl << "No exception was thrown." << std::endl << "The last binlog event: " << std::endl; - last_event->dump(std::cout); - std::cout << std::endl; + DB::WriteBufferFromOStream cout(std::cout); + cout << "Check passed. " << '\n' << "No exception was thrown." << '\n' << "The last binlog event: " << '\n'; + last_event->dump(cout); + cout << '\n'; return 0; } diff --git a/utils/check-style/check-style b/utils/check-style/check-style index f9818a1e2bb..56fd6843693 100755 --- a/utils/check-style/check-style +++ b/utils/check-style/check-style @@ -105,3 +105,6 @@ find $ROOT_PATH/{src,base,programs,utils} -name '*.h' -or -name '*.cpp' | xargs # Trailing whitespaces find $ROOT_PATH/{src,base,programs,utils} -name '*.h' -or -name '*.cpp' | xargs grep -P ' $' | grep -P '.' && echo "^ Trailing whitespaces." + +# Forbid stringstream because it's easy to use them incorrectly and hard to debug possible issues +find $ROOT_PATH/{src,programs,utils} -name '*.h' -or -name '*.cpp' | xargs grep 'std::[io]\?stringstream' | grep -v "STYLE_CHECK_ALLOW_STD_STRING_STREAM" && echo "Use WriteBufferFromOwnString or ReadBufferFromString instead of std::stringstream" diff --git a/utils/db-generator/query_db_generator.cpp b/utils/db-generator/query_db_generator.cpp index c8aae4a56f3..e3c5b862dbc 100644 --- a/utils/db-generator/query_db_generator.cpp +++ b/utils/db-generator/query_db_generator.cpp @@ -16,6 +16,7 @@ #include #include #include +#include #include #include @@ -828,9 +829,9 @@ FuncRet arithmeticFunc(DB::ASTPtr ch, std::map & columns) FuncRet r(ret_type, ""); if (no_indent) { - std::ostringstream ss; - formatAST(*ch, ss); - r.value = ss.str(); + DB::WriteBufferFromOwnString buf; + formatAST(*ch, buf); + r.value = buf.str(); } return r; } @@ -990,10 +991,10 @@ FuncRet simpleFunc(DB::ASTPtr ch, std::map & columns) { if (no_indent) { - std::ostringstream ss; - formatAST(*ch, ss); + DB::WriteBufferFromOwnString buf; + formatAST(*ch, buf); auto r = func_to_return_type[boost::algorithm::to_lower_copy(x->name)]; - r.value = ss.str(); + r.value = buf.str(); return r; } return func_to_return_type[boost::algorithm::to_lower_copy(x->name)]; @@ -1003,11 +1004,11 @@ FuncRet simpleFunc(DB::ASTPtr ch, std::map & columns) { if (no_indent) { - std::ostringstream ss; - formatAST(*ch, ss); + DB::WriteBufferFromOwnString buf; + formatAST(*ch, buf); return FuncRet( func_to_param_type[boost::algorithm::to_lower_copy(x->name)], - ss.str()); + buf.str()); } return FuncRet( func_to_param_type[boost::algorithm::to_lower_copy(x->name)], diff --git a/utils/list-versions/version_date.tsv b/utils/list-versions/version_date.tsv index 0e55de496b7..47f6aefe406 100644 --- a/utils/list-versions/version_date.tsv +++ b/utils/list-versions/version_date.tsv @@ -1,3 +1,4 @@ +v20.11.2.1-stable 2020-11-11 v20.10.3.30-stable 2020-10-29 v20.10.2.20-stable 2020-10-23 v20.9.4.76-stable 2020-10-29 diff --git a/utils/test-data-generator/ProtobufDelimitedMessagesSerializer.cpp b/utils/test-data-generator/ProtobufDelimitedMessagesSerializer.cpp index d16df83d12f..ad465913313 100644 --- a/utils/test-data-generator/ProtobufDelimitedMessagesSerializer.cpp +++ b/utils/test-data-generator/ProtobufDelimitedMessagesSerializer.cpp @@ -9,7 +9,7 @@ #include "00825_protobuf_format_syntax2.pb.h" -void writeInsertDataQueryForInputTest(std::stringstream & delimited_messages, const std::string & table_name, const std::string & format_schema, std::ostream & out) +void writeInsertDataQueryForInputTest(std::stringstream & delimited_messages, const std::string & table_name, const std::string & format_schema, std::ostream & out) // STYLE_CHECK_ALLOW_STD_STRING_STREAM { out << "echo -ne '"; std::string bytes = delimited_messages.str(); @@ -27,7 +27,7 @@ void writeInsertDataQueryForInputTest(std::stringstream & delimited_messages, co void writeInsertDataQueriesForInputTest(std::ostream & out) { - std::stringstream ss; + std::stringstream ss; // STYLE_CHECK_ALLOW_STD_STRING_STREAM { Person person; person.set_uuid("a7522158-3d41-4b77-ad69-6c598ee55c49"); diff --git a/utils/zookeeper-cli/zookeeper-cli.cpp b/utils/zookeeper-cli/zookeeper-cli.cpp index 17a8c9f0da8..47423dcd202 100644 --- a/utils/zookeeper-cli/zookeeper-cli.cpp +++ b/utils/zookeeper-cli/zookeeper-cli.cpp @@ -80,7 +80,7 @@ int main(int argc, char ** argv) try { - std::stringstream ss(line); + std::stringstream ss(line); // STYLE_CHECK_ALLOW_STD_STRING_STREAM std::string cmd; ss >> cmd;