diff --git a/CHANGELOG.md b/CHANGELOG.md index 87d478a62df..b090e541101 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,140 @@ +## ClickHouse release 19.1.6, 2019-01-24 + +### Backward Incompatible Change +* Removed `ALTER MODIFY PRIMARY KEY` command because it was superseded by the `ALTER MODIFY ORDER BY` command. [#3887](https://github.com/yandex/ClickHouse/pull/3887) ([ztlpn](https://github.com/ztlpn)) + +### New Features +* Add ability to choose per column codecs for storage log and tiny log. [#4111](https://github.com/yandex/ClickHouse/pull/4111) ([alesapin](https://github.com/alesapin)) +* Added functions `filesystemAvailable`, `filesystemFree`, `filesystemCapacity`. [#4097](https://github.com/yandex/ClickHouse/pull/4097) ([bgranvea](https://github.com/bgranvea)) +* Add custom compression codecs. [#3899](https://github.com/yandex/ClickHouse/pull/3899) ([alesapin](https://github.com/alesapin)) +* Added hashing functions `xxHash64` and `xxHash32`. [#3905](https://github.com/yandex/ClickHouse/pull/3905) ([filimonov](https://github.com/filimonov)) +* Added multiple joins emulation (very experimental). [#3946](https://github.com/yandex/ClickHouse/pull/3946) ([4ertus2](https://github.com/4ertus2)) +* Added support for CatBoost multiclass models evaluation. Function `modelEvaluate` returns tuple with per-class raw predictions for multiclass models. `libcatboostmodel.so` should be built with [#607](https://github.com/catboost/catboost/pull/607). [#3959](https://github.com/yandex/ClickHouse/pull/3959) ([KochetovNicolai](https://github.com/KochetovNicolai)) +* Added gccHash function which uses the same hash seed as [gcc](https://github.com/gcc-mirror/gcc/blob/41d6b10e96a1de98e90a7c0378437c3255814b16/libstdc%2B%2B-v3/include/bits/functional_hash.h#L191) [#4000](https://github.com/yandex/ClickHouse/pull/4000) ([sundy-li](https://github.com/sundy-li)) +* Added compression codec delta. [#4052](https://github.com/yandex/ClickHouse/pull/4052) ([alesapin](https://github.com/alesapin)) +* Added multi searcher to search from multiple constant strings from big haystack. Added functions (`multiPosition`, `multiSearch` ,`firstMatch`) * (` `, `UTF8`, `CaseInsensitive`, `CaseInsensitiveUTF8`) [#4053](https://github.com/yandex/ClickHouse/pull/4053) ([danlark1](https://github.com/danlark1)) +* Added ability to alter compression codecs. [#4054](https://github.com/yandex/ClickHouse/pull/4054) ([alesapin](https://github.com/alesapin)) +* Add ability to write data into HDFS and small refactoring. [#4084](https://github.com/yandex/ClickHouse/pull/4084) ([alesapin](https://github.com/alesapin)) +* Removed some redundant objects from compiled expressions cache (optimization). [#4042](https://github.com/yandex/ClickHouse/pull/4042) ([alesapin](https://github.com/alesapin)) +* Added functions `JavaHash`, `HiveHash`. [#3811](https://github.com/yandex/ClickHouse/pull/3811) ([shangshujie365](https://github.com/shangshujie365)) +* Added functions `left`, `right`, `trim`, `ltrim`, `rtrim`, `timestampadd`, `timestampsub`. [#3826](https://github.com/yandex/ClickHouse/pull/3826) ([blinkov](https://github.com/blinkov)) +* Added function `remoteSecure`. Function works as `remote`, but uses secure connection. [#4088](https://github.com/yandex/ClickHouse/pull/4088) ([proller](https://github.com/proller)) + +### Improvements +* Support for IF NOT EXISTS in ALTER TABLE ADD COLUMN statements, and for IF EXISTS in DROP/MODIFY/CLEAR/COMMENT COLUMN. [#3900](https://github.com/yandex/ClickHouse/pull/3900) ([bgranvea](https://github.com/bgranvea)) +* Function `parseDateTimeBestEffort`: support for formats `DD.MM.YYYY`, `DD.MM.YY`, `DD-MM-YYYY`, `DD-Mon-YYYY`, `DD/Month/YYYY` and similar. [#3922](https://github.com/yandex/ClickHouse/pull/3922) ([alexey-milovidov](https://github.com/alexey-milovidov)) +* Add a MergeTree setting `use_minimalistic_part_header_in_zookeeper`. If enabled, Replicated tables will store compact part metadata in a single part znode. This can dramatically reduce ZooKeeper snapshot size (especially if the tables have a lot of columns). Note that after enabling this setting you will not be able to downgrade to a version that doesn't support it. [#3960](https://github.com/yandex/ClickHouse/pull/3960) ([ztlpn](https://github.com/ztlpn)) +* Add an DFA-based implementation for functions `sequenceMatch` and `sequenceCount` in case pattern doesn't contain time. [#\](https://github.com/yandex/ClickHouse/pull/4004) ([ercolanelli-leo](https://github.com/ercolanelli-leo)) +* Changed the way CapnProtoInputStream creates actions in such a way that it now support structures that are jagged. [#4063](https://github.com/yandex/ClickHouse/pull/4063) ([Miniwoffer](https://github.com/Miniwoffer)) +* Better way to collect columns, tables and joins from AST when checking required columns. [#3930](https://github.com/yandex/ClickHouse/pull/3930) ([4ertus2](https://github.com/4ertus2)) +* Zero left padding PODArray so that -1 element is always valid and zeroed. It's used for branchless Offset access. [#3920](https://github.com/yandex/ClickHouse/pull/3920) ([amosbird](https://github.com/amosbird)) +* Performance improvement for int serialization. [#3968](https://github.com/yandex/ClickHouse/pull/3968) ([amosbird](https://github.com/amosbird)) +* Moved debian/ specific entries to debian/.gitignore [#4106](https://github.com/yandex/ClickHouse/pull/4106) ([gerasiov](https://github.com/gerasiov)) +* Decreased the number of connections in case of large number of Distributed tables in a single server. [#3726](https://github.com/yandex/ClickHouse/pull/3726) ([zhang2014](https://github.com/zhang2014)) +* Supported totals row for `WITH TOTALS` query for ODBC driver (ODBCDriver2 format). [#3836](https://github.com/yandex/ClickHouse/pull/3836) ([nightweb](https://github.com/nightweb)) +* Better constant expression folding. Possibility to skip unused shards if SELECT query filters by sharding_key (setting `distributed_optimize_skip_select_on_unused_shards`). [#3851](https://github.com/yandex/ClickHouse/pull/3851) ([abyss7](https://github.com/abyss7)) +* Do not log from odbc-bridge when there is no console. [#3857](https://github.com/yandex/ClickHouse/pull/3857) ([alesapin](https://github.com/alesapin)) +* Forbid using aggregate functions inside scalar subqueries. [#3865](https://github.com/yandex/ClickHouse/pull/3865) ([abyss7](https://github.com/abyss7)) +* Added ability to use Enums as integers inside if function. [#3875](https://github.com/yandex/ClickHouse/pull/3875) ([abyss7](https://github.com/abyss7)) +* Added `low_cardinality_allow_in_native_format` setting. If disabled, do not use `LowCadrinality` type in native format. [#3879](https://github.com/yandex/ClickHouse/pull/3879) ([KochetovNicolai](https://github.com/KochetovNicolai)) +* Removed duplicate code. [#3915](https://github.com/yandex/ClickHouse/pull/3915) ([sergey-v-galtsev](https://github.com/sergey-v-galtsev)) +* Minor improvements in StorageKafka. [#3919](https://github.com/yandex/ClickHouse/pull/3919) ([alexey-milovidov](https://github.com/alexey-milovidov)) +* Automatically disable logs in negative tests. [#3940](https://github.com/yandex/ClickHouse/pull/3940) ([4ertus2](https://github.com/4ertus2)) +* Refactored SyntaxAnalyzer. [#4014](https://github.com/yandex/ClickHouse/pull/4014) ([4ertus2](https://github.com/4ertus2)) +* Reverted jemalloc patch which lead to performance degradation. [#4018](https://github.com/yandex/ClickHouse/pull/4018) ([alexey-milovidov](https://github.com/alexey-milovidov)) +* Refactored QueryNormalizer. Unified column sources for ASTIdentifier and ASTQualifiedAsterisk (were different), removed column duplicates for ASTQualifiedAsterisk sources, cleared asterisks replacement. [#4031](https://github.com/yandex/ClickHouse/pull/4031) ([4ertus2](https://github.com/4ertus2)) +* Refactored code with ASTIdentifier. [#4056](https://github.com/yandex/ClickHouse/pull/4056) [#4077](https://github.com/yandex/ClickHouse/pull/4077) [#4087](https://github.com/yandex/ClickHouse/pull/4087) ([4ertus2](https://github.com/4ertus2)) +* Improve error message in `clickhouse-test` script when no ClickHouse binary was found. [#4130](https://github.com/yandex/ClickHouse/pull/4130) ([Miniwoffer](https://github.com/Miniwoffer)) +* Rewrited code to calculate integer conversion function monotonicity. [#3921](https://github.com/yandex/ClickHouse/pull/3921) ([alexey-milovidov](https://github.com/alexey-milovidov)) +* Fixed typos in comments. [#4089](https://github.com/yandex/ClickHouse/pull/4089) ([kvinty](https://github.com/kvinty)) + +### Build/Testing/Packaging Improvements +* Added minimal support for powerpc build. [#4132](https://github.com/yandex/ClickHouse/pull/4132) ([danlark1](https://github.com/danlark1)) +* Fixed error when the server cannot start with the `bash: /usr/bin/clickhouse-extract-from-config: Operation not permitted` message within Docker or systemd-nspawn. [#4136](https://github.com/yandex/ClickHouse/pull/4136) ([alexey-milovidov](https://github.com/alexey-milovidov)) +* Updated `mariadb-client` library. Fixed one of issues found by UBSan. [#3924](https://github.com/yandex/ClickHouse/pull/3924) ([alexey-milovidov](https://github.com/alexey-milovidov)) +* Some fixes for UBSan builds. [#3926](https://github.com/yandex/ClickHouse/pull/3926) [#3948](https://github.com/yandex/ClickHouse/pull/3948) ([alexey-milovidov](https://github.com/alexey-milovidov)) +* Move docker images to 18.10 and add compatibility file for glibc >= 2.28 [#3965](https://github.com/yandex/ClickHouse/pull/3965) ([alesapin](https://github.com/alesapin)) +* Add env variable if user don't want to chown directories in server docker image. [#3967](https://github.com/yandex/ClickHouse/pull/3967) ([alesapin](https://github.com/alesapin)) +* Stateful functional tests are run on public available dataset. [#3969](https://github.com/yandex/ClickHouse/pull/3969) ([alexey-milovidov](https://github.com/alexey-milovidov)) +* Enabled most of the warnings from `-Weverything` in clang. Enabled `-Wpedantic`. [#3986](https://github.com/yandex/ClickHouse/pull/3986) ([alexey-milovidov](https://github.com/alexey-milovidov)) +* Link to libLLVM rather than to individual LLVM libs when USE_STATIC_LIBRARIES is off. [#3989](https://github.com/yandex/ClickHouse/pull/3989) ([orivej](https://github.com/orivej)) +* Added a few more warnings that are available only in clang 8. [#3993](https://github.com/yandex/ClickHouse/pull/3993) ([alexey-milovidov](https://github.com/alexey-milovidov)) +* Fixed bugs found by PVS-Studio. [#4013](https://github.com/yandex/ClickHouse/pull/4013) ([alexey-milovidov](https://github.com/alexey-milovidov)) +* Added sanitizer variables for test images. [#4072](https://github.com/yandex/ClickHouse/pull/4072) ([alesapin](https://github.com/alesapin)) +* clickhouse-server debian package will recommend `libcap2-bin` package to use `setcap` tool for setting capabilities. This is optional. [#4093](https://github.com/yandex/ClickHouse/pull/4093) ([alexey-milovidov](https://github.com/alexey-milovidov)) +* Improved compilation time, fixed includes. [#3898](https://github.com/yandex/ClickHouse/pull/3898) ([proller](https://github.com/proller)) +* Added performance tests for hash functions. [#3918](https://github.com/yandex/ClickHouse/pull/3918) ([filimonov](https://github.com/filimonov)) +* Fixed cyclic library dependences. [#3958](https://github.com/yandex/ClickHouse/pull/3958) ([proller](https://github.com/proller)) +* Improved compilation with low available memory. [#4030](https://github.com/yandex/ClickHouse/pull/4030) ([proller](https://github.com/proller)) + +### Bug Fixes +* Fix bug when in remote table function execution when wrong restrictions were used for in `getStructureOfRemoteTable`. [#4009](https://github.com/yandex/ClickHouse/pull/4009) ([alesapin](https://github.com/alesapin)) +* Fix a leak of netlink sockets. They were placed in a pool where they were never deleted and new sockets were created at the start of a new thread when all current sockets were in use. [#4017](https://github.com/yandex/ClickHouse/pull/4017) ([ztlpn](https://github.com/ztlpn)) +* Regression in master. Fix "Unknown identifier" error in case column names appear in lambdas. [#4115](https://github.com/yandex/ClickHouse/pull/4115) ([4ertus2](https://github.com/4ertus2)) +* Fix bug with closing /proc/self/fd earlier than all fds were read from /proc. [#4120](https://github.com/yandex/ClickHouse/pull/4120) ([alesapin](https://github.com/alesapin)) +* Fixed misspells in **comments** and **string literals** under `dbms`. [#4122](https://github.com/yandex/ClickHouse/pull/4122) ([maiha](https://github.com/maiha)) +* Fixed String to UInt monotonic conversion in case of usage String in primary key. [#3870](https://github.com/yandex/ClickHouse/pull/3870) ([zhang2014](https://github.com/zhang2014)) +* Add checking that 'SET send_logs_level = value' query accept appropriate value. [#3873](https://github.com/yandex/ClickHouse/pull/3873) ([s-mx](https://github.com/s-mx)) +* Fixed a race condition when executing a distributed ALTER task. The race condition led to more than one replica trying to execute the task and all replicas except one failing with a ZooKeeper error. [#3904](https://github.com/yandex/ClickHouse/pull/3904) ([ztlpn](https://github.com/ztlpn)) +* Fixed segfault in `arrayEnumerateUniq`, `arrayEnumerateDense` functions in case of some invalid arguments. [#3909](https://github.com/yandex/ClickHouse/pull/3909) ([alexey-milovidov](https://github.com/alexey-milovidov)) +* Fix UB in StorageMerge. [#3910](https://github.com/yandex/ClickHouse/pull/3910) ([amosbird](https://github.com/amosbird)) +* Fixed segfault in functions `addDays`, `subtractDays`. [#3913](https://github.com/yandex/ClickHouse/pull/3913) ([alexey-milovidov](https://github.com/alexey-milovidov)) +* Fixed error: functions `round`, `floor`, `trunc`, `ceil` may return bogus result when executed on integer argument and large negative scale. [#3914](https://github.com/yandex/ClickHouse/pull/3914) ([alexey-milovidov](https://github.com/alexey-milovidov)) +* Fixed a bug introduced by 'kill query sync' which leads to a core dump. [#3916](https://github.com/yandex/ClickHouse/pull/3916) ([fancyqlx](https://github.com/fancyqlx)) +* Fix bug with long delay after empty replication queue. [#3928](https://github.com/yandex/ClickHouse/pull/3928) ([alesapin](https://github.com/alesapin)) +* Don't do exponential backoff when there is nothing to do for task. [#3932](https://github.com/yandex/ClickHouse/pull/3932) ([alesapin](https://github.com/alesapin)) +* Fix a bug that led to hangups in threads that perform ALTERs of Replicated tables and in the thread that updates configuration from ZooKeeper. #2947 #3891 [#3934](https://github.com/yandex/ClickHouse/pull/3934) ([ztlpn](https://github.com/ztlpn)) +* Fixed error in internal implementation of `quantileTDigest` (found by Artem Vakhrushev). This error never happens in ClickHouse and was relevant only for those who use ClickHouse codebase as a library directly. [#3935](https://github.com/yandex/ClickHouse/pull/3935) ([alexey-milovidov](https://github.com/alexey-milovidov)) +* Fix bug with wrong prefix for ipv4 subnet masks. [#3945](https://github.com/yandex/ClickHouse/pull/3945) ([alesapin](https://github.com/alesapin)) +* Fix a bug when `from_zk` config elements weren't refreshed after a request to ZooKeeper timed out. #2947 [#3947](https://github.com/yandex/ClickHouse/pull/3947) ([ztlpn](https://github.com/ztlpn)) +* Fixed dictionary copying at LowCardinality::cloneEmpty() method which lead to excessive memory usage in case of inserting into table with LowCardinality primary key. [#3955](https://github.com/yandex/ClickHouse/pull/3955) ([KochetovNicolai](https://github.com/KochetovNicolai)) +* Fixed crash (`std::terminate`) in rare cases when a new thread cannot be created due to exhausted resources. [#3956](https://github.com/yandex/ClickHouse/pull/3956) ([alexey-milovidov](https://github.com/alexey-milovidov)) +* Fix user and password forwarding for replicated tables queries. [#3957](https://github.com/yandex/ClickHouse/pull/3957) ([alesapin](https://github.com/alesapin)) +* Fixed very rare race condition that can happen when listing tables in Dictionary database while reloading dictionaries. [#3970](https://github.com/yandex/ClickHouse/pull/3970) ([alexey-milovidov](https://github.com/alexey-milovidov)) +* Fixed LowCardinality serialization for Native format in case of empty arrays. #3907 [#4011](https://github.com/yandex/ClickHouse/pull/4011) ([KochetovNicolai](https://github.com/KochetovNicolai)) +* Fixed incorrect result while using distinct by single LowCardinality numeric column. #3895 [#4012](https://github.com/yandex/ClickHouse/pull/4012) ([KochetovNicolai](https://github.com/KochetovNicolai)) +* Make compiled_expression_cache_size setting limited by default. [#4041](https://github.com/yandex/ClickHouse/pull/4041) ([alesapin](https://github.com/alesapin)) +* Fix ubsan bug in compression codecs. [#4069](https://github.com/yandex/ClickHouse/pull/4069) ([alesapin](https://github.com/alesapin)) +* Allow Kafka Engine to ignore some number of parsing errors per block. [#4094](https://github.com/yandex/ClickHouse/pull/4094) ([abyss7](https://github.com/abyss7)) +* Fixed glibc compatibility issues. [#4100](https://github.com/yandex/ClickHouse/pull/4100) ([alexey-milovidov](https://github.com/alexey-milovidov)) +* Fixed issues found by PVS-Studio. [#4103](https://github.com/yandex/ClickHouse/pull/4103) ([alexey-milovidov](https://github.com/alexey-milovidov)) +* Fix a way how to collect array join columns. [#4121](https://github.com/yandex/ClickHouse/pull/4121) ([4ertus2](https://github.com/4ertus2)) +* Fixed incorrect result when HAVING was used with ROLLUP or CUBE. [#3756](https://github.com/yandex/ClickHouse/issues/3756) [#3837](https://github.com/yandex/ClickHouse/pull/3837) ([reflection](https://github.com/reflection)) +* Fixed specialized aggregation with LowCardinality key (in case when `compile` setting is enabled). [#3886](https://github.com/yandex/ClickHouse/pull/3886) ([KochetovNicolai](https://github.com/KochetovNicolai)) +* Fixed data type check in type conversion functions. [#3896](https://github.com/yandex/ClickHouse/pull/3896) ([zhang2014](https://github.com/zhang2014)) +* Fixed column aliases for query with `JOIN ON` syntax and distributed tables. [#3980](https://github.com/yandex/ClickHouse/pull/3980) ([zhang2014](https://github.com/zhang2014)) +* Fixed issues detected by UBSan. [#3021](https://github.com/yandex/ClickHouse/pull/3021) ([alexey-milovidov](https://github.com/alexey-milovidov)) + +### Doc fixes +* Translated table engines related part to Chinese. [#3844](https://github.com/yandex/ClickHouse/pull/3844) ([lamber-ken](https://github.com/lamber-ken)) +* Fixed `toStartOfFiveMinute` description. [#4096](https://github.com/yandex/ClickHouse/pull/4096) ([cheesedosa](https://github.com/cheesedosa)) +* Added description for client `--secure` argument. [#3961](https://github.com/yandex/ClickHouse/pull/3961) ([vicdashkov](https://github.com/vicdashkov)) +* Added descriptions for settings `merge_tree_uniform_read_distribution`, `merge_tree_min_rows_for_concurrent_read`, `merge_tree_min_rows_for_seek`, `merge_tree_coarse_index_granularity`, `merge_tree_max_rows_to_use_cache` [#4024](https://github.com/yandex/ClickHouse/pull/4024) ([BayoNet](https://github.com/BayoNet)) +* Minor doc fixes. [#4098](https://github.com/yandex/ClickHouse/pull/4098) ([blinkov](https://github.com/blinkov)) +* Updated example for zookeeper config setting. [#3883](https://github.com/yandex/ClickHouse/pull/3883) [#3894](https://github.com/yandex/ClickHouse/pull/3894) ([ogorbacheva](https://github.com/ogorbacheva)) +* Updated info about escaping in formats Vertical, Pretty and VerticalRaw. [#4118](https://github.com/yandex/ClickHouse/pull/4118) ([ogorbacheva](https://github.com/ogorbacheva)) +* Adding description of the functions for working with UUID. [#4059](https://github.com/yandex/ClickHouse/pull/4059) ([ogorbacheva](https://github.com/ogorbacheva)) +* Add the description of the CHECK TABLE query. [#3881](https://github.com/yandex/ClickHouse/pull/3881) [#4043](https://github.com/yandex/ClickHouse/pull/4043) ([ogorbacheva](https://github.com/ogorbacheva)) +* Add `zh/tests` doc translate to Chinese. [#4034](https://github.com/yandex/ClickHouse/pull/4034) ([sundy-li](https://github.com/sundy-li)) +* Added documentation about functions `multiPosition`, `firstMatch`, `multiSearch`. [#4123](https://github.com/yandex/ClickHouse/pull/4123) ([danlark1](https://github.com/danlark1)) +* Add puppet module to the list of the third party libraries. [#3862](https://github.com/yandex/ClickHouse/pull/3862) ([Felixoid](https://github.com/Felixoid)) +* Fixed typo in the English version of Creating a Table example [#3872](https://github.com/yandex/ClickHouse/pull/3872) ([areldar](https://github.com/areldar)) +* Mention about nagios plugin for ClickHouse [#3878](https://github.com/yandex/ClickHouse/pull/3878) ([lisuml](https://github.com/lisuml)) +* Update of query language syntax description. [#4065](https://github.com/yandex/ClickHouse/pull/4065) ([BayoNet](https://github.com/BayoNet)) +* Added documentation for per-column compression codecs. [#4073](https://github.com/yandex/ClickHouse/pull/4073) ([alex-krash](https://github.com/alex-krash)) +* Updated articles about CollapsingMergeTree, GraphiteMergeTree, Replicated*MergeTree, `CREATE TABLE` query [#4085](https://github.com/yandex/ClickHouse/pull/4085) ([BayoNet](https://github.com/BayoNet)) +* Other minor improvements. [#3897](https://github.com/yandex/ClickHouse/pull/3897) [#3923](https://github.com/yandex/ClickHouse/pull/3923) [#4066](https://github.com/yandex/ClickHouse/pull/4066) [#3860](https://github.com/yandex/ClickHouse/pull/3860) [#3906](https://github.com/yandex/ClickHouse/pull/3906) [#3936](https://github.com/yandex/ClickHouse/pull/3936) [#3975](https://github.com/yandex/ClickHouse/pull/3975) ([ogorbacheva](https://github.com/ogorbacheva)) ([ogorbacheva](https://github.com/ogorbacheva)) ([ogorbacheva](https://github.com/ogorbacheva)) ([blinkov](https://github.com/blinkov)) ([blinkov](https://github.com/blinkov)) ([sdk2](https://github.com/sdk2)) ([blinkov](https://github.com/blinkov)) + +### Other +* Updated librdkafka to v1.0.0-RC5. Used cppkafka instead of raw C interface. [#4025](https://github.com/yandex/ClickHouse/pull/4025) ([abyss7](https://github.com/abyss7)) +* Fixed `hidden` on page title [#4033](https://github.com/yandex/ClickHouse/pull/4033) ([xboston](https://github.com/xboston)) +* Updated year in copyright to 2019. [#4039](https://github.com/yandex/ClickHouse/pull/4039) ([xboston](https://github.com/xboston)) +* Added check that server process is started from the data directory's owner. Do not start server from root. [#3785](https://github.com/yandex/ClickHouse/pull/3785) ([sergey-v-galtsev](https://github.com/sergey-v-galtsev)) +* Removed function `shardByHash`. [#3833](https://github.com/yandex/ClickHouse/pull/3833) ([alexey-milovidov](https://github.com/alexey-milovidov)) +* Fixed typo in ClusterCopier. [#3854](https://github.com/yandex/ClickHouse/pull/3854) ([dqminh](https://github.com/dqminh)) +* Minor grammar fixes. [#3855](https://github.com/yandex/ClickHouse/pull/3855) ([intgr](https://github.com/intgr)) +* Added test script to reproduce performance degradation in jemalloc. [#4036](https://github.com/yandex/ClickHouse/pull/4036) ([alexey-milovidov](https://github.com/alexey-milovidov)) + ## ClickHouse release 18.16.1, 2018-12-21 ### Bug fixes: diff --git a/CMakeLists.txt b/CMakeLists.txt index 4265cc126f1..98c3643f055 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -81,7 +81,7 @@ option (ENABLE_TESTS "Enables tests" ON) if (CMAKE_SYSTEM_PROCESSOR MATCHES "amd64|x86_64") option (USE_INTERNAL_MEMCPY "Use internal implementation of 'memcpy' function instead of provided by libc. Only for x86_64." ON) - if (OS_LINUX AND NOT UNBUNDLED) + if (OS_LINUX AND NOT UNBUNDLED AND MAKE_STATIC_LIBRARIES) option (GLIBC_COMPATIBILITY "Set to TRUE to enable compatibility with older glibc libraries. Only for x86_64, Linux. Implies USE_INTERNAL_MEMCPY." ON) if (GLIBC_COMPATIBILITY) message (STATUS "Some symbols from glibc will be replaced for compatibility") @@ -232,6 +232,7 @@ include (cmake/find_llvm.cmake) include (cmake/find_cpuid.cmake) include (cmake/find_libgsasl.cmake) include (cmake/find_libxml2.cmake) +include (cmake/find_protobuf.cmake) include (cmake/find_hdfs3.cmake) include (cmake/find_consistent-hashing.cmake) include (cmake/find_base64.cmake) diff --git a/README.md b/README.md index 0e9974f763f..f496e32b905 100644 --- a/README.md +++ b/README.md @@ -10,3 +10,7 @@ ClickHouse is an open-source column-oriented database management system that all * [Blog](https://clickhouse.yandex/blog/en/) contains various ClickHouse-related articles, as well as announces and reports about events. * [Contacts](https://clickhouse.yandex/#contacts) can help to get your questions answered if there are any. * You can also [fill this form](https://forms.yandex.com/surveys/meet-yandex-clickhouse-team/) to meet Yandex ClickHouse team in person. + +## Upcoming Events + +* [C++ ClickHouse and CatBoost Sprints](https://events.yandex.ru/events/ClickHouse/2-feb-2019/) in Moscow on February 2. diff --git a/ci/README.md b/ci/README.md index 733cbce80c9..003f547fb79 100644 --- a/ci/README.md +++ b/ci/README.md @@ -25,16 +25,17 @@ Various possible options. We are not going to automate testing all of them. #### CPU architectures: - x86_64; -- AArch64. +- AArch64; +- PowerPC64LE. -x86_64 is the main CPU architecture. We also have minimal support for AArch64. +x86_64 is the main CPU architecture. We also have minimal support for AArch64 and PowerPC64LE. #### Operating systems: - Linux; - FreeBSD. -We also target Mac OS X, but it's more difficult to test. -Linux is the main. FreeBSD is also supported as production OS. +We also target Mac OS X, but it's more difficult to test. +Linux is the main. FreeBSD is also supported as production OS. Mac OS is intended only for development and have minimal support: client should work, server should just start. #### Linux distributions: @@ -98,14 +99,14 @@ We also have intent to build RPM and simple tgz packages. - from contrib directory (submodules); - from OS packages. -The only production option is to use libraries from contrib directory. +The only production option is to use libraries from contrib directory. Using libraries from OS packages is discouraged, but we also support this option. #### Linkage types: - static; - shared; -Static linking is the only option for production usage. +Static linking is the only option for production usage. We also have support for shared linking, but it is indended only for developers. #### Make tools: diff --git a/cmake/arch.cmake b/cmake/arch.cmake index abc30d99e32..f8e18629b09 100644 --- a/cmake/arch.cmake +++ b/cmake/arch.cmake @@ -24,3 +24,10 @@ if (CMAKE_CXX_COMPILER_ID STREQUAL "GNU") elseif (CMAKE_CXX_COMPILER_ID STREQUAL "Clang") set (COMPILER_CLANG 1) endif () + +if (CMAKE_SYSTEM_PROCESSOR MATCHES "^(ppc64le.*|PPC64LE.*)") + set (ARCH_PPC64LE 1) + if (COMPILER_CLANG OR (COMPILER_GCC AND CMAKE_CXX_COMPILER_VERSION VERSION_LESS 8)) + message(FATAL_ERROR "Only gcc-8 is supported for powerpc architecture") + endif () +endif () diff --git a/cmake/find_gtest.cmake b/cmake/find_gtest.cmake index ce0a3d32785..19562e8d2ea 100644 --- a/cmake/find_gtest.cmake +++ b/cmake/find_gtest.cmake @@ -15,6 +15,11 @@ endif () if (NOT GTEST_INCLUDE_DIRS AND NOT MISSING_INTERNAL_GTEST_LIBRARY) set (USE_INTERNAL_GTEST_LIBRARY 1) set (GTEST_MAIN_LIBRARIES gtest_main) + set (GTEST_INCLUDE_DIRS ${ClickHouse_SOURCE_DIR}/contrib/googletest/googletest) endif () -message (STATUS "Using gtest: ${GTEST_INCLUDE_DIRS} : ${GTEST_MAIN_LIBRARIES}") +if(GTEST_INCLUDE_DIRS AND GTEST_MAIN_LIBRARIES) + set(USE_GTEST 1) +endif() + +message (STATUS "Using gtest=${USE_GTEST}: ${GTEST_INCLUDE_DIRS} : ${GTEST_MAIN_LIBRARIES}") diff --git a/cmake/find_protobuf.cmake b/cmake/find_protobuf.cmake index 5daf5a0c186..03904ef7973 100644 --- a/cmake/find_protobuf.cmake +++ b/cmake/find_protobuf.cmake @@ -1,18 +1,29 @@ -option (USE_INTERNAL_PROTOBUF_LIBRARY "Set to FALSE to use system protobuf instead of bundled" ON) +option(USE_INTERNAL_PROTOBUF_LIBRARY "Set to FALSE to use system protobuf instead of bundled" ${NOT_UNBUNDLED}) -if (NOT USE_INTERNAL_PROTOBUF_LIBRARY) +if(NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/protobuf/cmake/CMakeLists.txt") + if(USE_INTERNAL_PROTOBUF_LIBRARY) + message(WARNING "submodule contrib/protobuf is missing. to fix try run: \n git submodule update --init --recursive") + set(USE_INTERNAL_PROTOBUF_LIBRARY 0) + endif() + set(MISSING_INTERNAL_PROTOBUF_LIBRARY 1) +endif() + +if(NOT USE_INTERNAL_PROTOBUF_LIBRARY) find_package(Protobuf) -endif () +endif() if (Protobuf_LIBRARY AND Protobuf_INCLUDE_DIR) -else () - set(Protobuf_INCLUDE_DIR ${CMAKE_SOURCE_DIR}/contrib/protobuf/src) + set(USE_PROTOBUF 1) +elseif(NOT MISSING_INTERNAL_PROTOBUF_LIBRARY) + set(Protobuf_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/contrib/protobuf/src) + set(USE_PROTOBUF 1) + set(USE_INTERNAL_PROTOBUF_LIBRARY 1) set(Protobuf_LIBRARY libprotobuf) set(Protobuf_PROTOC_LIBRARY libprotoc) set(Protobuf_LITE_LIBRARY libprotobuf-lite) - set(Protobuf_PROTOC_EXECUTABLE ${CMAKE_BINARY_DIR}/contrib/protobuf/cmake/protoc) + set(Protobuf_PROTOC_EXECUTABLE ${ClickHouse_BINARY_DIR}/contrib/protobuf/cmake/protoc) if(NOT DEFINED PROTOBUF_GENERATE_CPP_APPEND_PATH) set(PROTOBUF_GENERATE_CPP_APPEND_PATH TRUE) @@ -77,4 +88,4 @@ else () endfunction() endif() -message (STATUS "Using protobuf: ${Protobuf_INCLUDE_DIR} : ${Protobuf_LIBRARY}") +message(STATUS "Using protobuf=${USE_PROTOBUF}: ${Protobuf_INCLUDE_DIR} : ${Protobuf_LIBRARY}") diff --git a/cmake/find_rdkafka.cmake b/cmake/find_rdkafka.cmake index 52a865cd50b..793bcd0d2d0 100644 --- a/cmake/find_rdkafka.cmake +++ b/cmake/find_rdkafka.cmake @@ -2,6 +2,11 @@ if (NOT ARCH_ARM AND NOT ARCH_32 AND NOT APPLE) option (ENABLE_RDKAFKA "Enable kafka" ON) endif () +if (NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/cppkafka/CMakeLists.txt") + message (WARNING "submodule contrib/cppkafka is missing. to fix try run: \n git submodule update --init --recursive") + set (ENABLE_RDKAFKA 0) +endif () + if (ENABLE_RDKAFKA) if (OS_LINUX AND NOT ARCH_ARM) diff --git a/cmake/find_zstd.cmake b/cmake/find_zstd.cmake index 0e6db94a7c0..24bc851ed57 100644 --- a/cmake/find_zstd.cmake +++ b/cmake/find_zstd.cmake @@ -14,6 +14,7 @@ if (ZSTD_LIBRARY AND ZSTD_INCLUDE_DIR) else () set (USE_INTERNAL_ZSTD_LIBRARY 1) set (ZSTD_LIBRARY zstd) + set (ZSTD_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/contrib/zstd/lib) endif () message (STATUS "Using zstd: ${ZSTD_INCLUDE_DIR} : ${ZSTD_LIBRARY}") diff --git a/cmake/test_cpu.cmake b/cmake/test_cpu.cmake index c360de5b962..42f9bac936e 100644 --- a/cmake/test_cpu.cmake +++ b/cmake/test_cpu.cmake @@ -27,6 +27,9 @@ if (HAVE_SSE41) set (COMPILER_FLAGS "${COMPILER_FLAGS} ${TEST_FLAG}") endif () +if (ARCH_PPC64LE) + set (COMPILER_FLAGS "${COMPILER_FLAGS} -maltivec -D__SSE2__=1 -DNO_WARN_X86_INTRINSICS") +endif () # gcc -dM -E -msse4.2 - < /dev/null | sort > gcc-dump-sse42 #define __SSE4_2__ 1 diff --git a/contrib/CMakeLists.txt b/contrib/CMakeLists.txt index 7b66bfce063..0c4b6c15287 100644 --- a/contrib/CMakeLists.txt +++ b/contrib/CMakeLists.txt @@ -207,14 +207,14 @@ if (USE_INTERNAL_LIBXML2_LIBRARY) add_subdirectory(libxml2-cmake) endif () +if (USE_INTERNAL_PROTOBUF_LIBRARY) + set(protobuf_BUILD_TESTS OFF CACHE INTERNAL "" FORCE) + set(protobuf_BUILD_SHARED_LIBS OFF CACHE INTERNAL "" FORCE) + set(protobuf_WITH_ZLIB 0 CACHE INTERNAL "" FORCE) # actually will use zlib, but skip find + add_subdirectory(protobuf/cmake) +endif () + if (USE_INTERNAL_HDFS3_LIBRARY) - include(${ClickHouse_SOURCE_DIR}/cmake/find_protobuf.cmake) - if (USE_INTERNAL_PROTOBUF_LIBRARY) - set(protobuf_BUILD_TESTS OFF CACHE INTERNAL "" FORCE) - set(protobuf_BUILD_SHARED_LIBS OFF CACHE INTERNAL "" FORCE) - set(protobuf_WITH_ZLIB 0 CACHE INTERNAL "" FORCE) # actually will use zlib, but skip find - add_subdirectory(protobuf/cmake) - endif () add_subdirectory(libhdfs3-cmake) endif () diff --git a/contrib/libdivide/libdivide.h b/contrib/libdivide/libdivide.h index dd3b85008c5..eaeaec7db6b 100644 --- a/contrib/libdivide/libdivide.h +++ b/contrib/libdivide/libdivide.h @@ -341,6 +341,13 @@ static inline __m128i libdivide_get_0000FFFF(void) { #pragma clang diagnostic pop #endif +/// This is a bug in gcc-8, _MM_SHUFFLE was forgotten, though in trunk it is ok https://github.com/gcc-mirror/gcc/blob/master/gcc/config/rs6000/xmmintrin.h#L61 +#if defined(__PPC__) +#ifndef _MM_SHUFFLE +#define _MM_SHUFFLE(w,x,y,z) (((w) << 6) | ((x) << 4) | ((y) << 2) | (z)) +#endif +#endif + static inline __m128i libdivide_s64_signbits(__m128i v) { //we want to compute v >> 63, that is, _mm_srai_epi64(v, 63). But there is no 64 bit shift right arithmetic instruction in SSE2. So we have to fake it by first duplicating the high 32 bit values, and then using a 32 bit shift. Another option would be to use _mm_srli_epi64(v, 63) and then subtract that from 0, but that approach appears to be substantially slower for unknown reasons __m128i hiBitsDuped = _mm_shuffle_epi32(v, _MM_SHUFFLE(3, 3, 1, 1)); diff --git a/contrib/libhdfs3-cmake/CMake/Options.cmake b/contrib/libhdfs3-cmake/CMake/Options.cmake index 728aef60e17..d7ccc8b6475 100644 --- a/contrib/libhdfs3-cmake/CMake/Options.cmake +++ b/contrib/libhdfs3-cmake/CMake/Options.cmake @@ -7,9 +7,9 @@ CHECK_FUNCTION_EXISTS(nanosleep HAVE_NANOSLEEP) SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fno-strict-aliasing") SET(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fno-strict-aliasing") -IF(ENABLE_SSE STREQUAL ON) +IF(ENABLE_SSE STREQUAL ON AND NOT ARCH_PPC64LE AND NOT ARCH_AARCH64 AND NOT ARCH_ARM) SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -msse4.2") -ENDIF(ENABLE_SSE STREQUAL ON) +ENDIF(ENABLE_SSE STREQUAL ON AND NOT ARCH_PPC64LE AND NOT ARCH_AARCH64 AND NOT ARCH_ARM) IF(NOT TEST_HDFS_PREFIX) SET(TEST_HDFS_PREFIX "./" CACHE STRING "default directory prefix used for test." FORCE) diff --git a/contrib/librdkafka-cmake/CMakeLists.txt b/contrib/librdkafka-cmake/CMakeLists.txt index 115c916e9f4..62337f60fb5 100644 --- a/contrib/librdkafka-cmake/CMakeLists.txt +++ b/contrib/librdkafka-cmake/CMakeLists.txt @@ -2,6 +2,7 @@ set(RDKAFKA_SOURCE_DIR ${CMAKE_SOURCE_DIR}/contrib/librdkafka/src) set(SRCS ${RDKAFKA_SOURCE_DIR}/crc32c.c + ${RDKAFKA_SOURCE_DIR}/rdkafka_zstd.c ${RDKAFKA_SOURCE_DIR}/rdaddr.c ${RDKAFKA_SOURCE_DIR}/rdavl.c ${RDKAFKA_SOURCE_DIR}/rdbuf.c @@ -59,5 +60,6 @@ set(SRCS add_library(rdkafka ${LINK_MODE} ${SRCS}) target_include_directories(rdkafka SYSTEM PUBLIC include) -target_include_directories(rdkafka SYSTEM PUBLIC ${RDKAFKA_SOURCE_DIR}) -target_link_libraries(rdkafka PUBLIC ${ZLIB_LIBRARIES} ${OPENSSL_SSL_LIBRARY} ${OPENSSL_CRYPTO_LIBRARY}) +target_include_directories(rdkafka SYSTEM PUBLIC ${RDKAFKA_SOURCE_DIR}) # Because weird logic with "include_next" is used. +target_include_directories(rdkafka SYSTEM PRIVATE ${ZSTD_INCLUDE_DIR}/common) # Because wrong path to "zstd_errors.h" is used. +target_link_libraries(rdkafka PUBLIC ${ZLIB_LIBRARIES} ${ZSTD_LIBRARY} ${OPENSSL_SSL_LIBRARY} ${OPENSSL_CRYPTO_LIBRARY}) diff --git a/contrib/librdkafka-cmake/config.h b/contrib/librdkafka-cmake/config.h index 2ffc5a497ae..266baae8dae 100644 --- a/contrib/librdkafka-cmake/config.h +++ b/contrib/librdkafka-cmake/config.h @@ -51,6 +51,8 @@ //#define WITH_PLUGINS 1 // zlib #define WITH_ZLIB 1 +// zstd +#define WITH_ZSTD 1 // WITH_SNAPPY #define WITH_SNAPPY 1 // WITH_SOCKEM @@ -60,7 +62,9 @@ // WITH_SASL_SCRAM //#define WITH_SASL_SCRAM 1 // crc32chw +#if !defined(__PPC__) #define WITH_CRC32C_HW 1 +#endif // regex #define HAVE_REGEX 1 // strndup diff --git a/contrib/libunwind/CMakeLists.txt b/contrib/libunwind/CMakeLists.txt index 47032be92bb..80bd8a301f0 100644 --- a/contrib/libunwind/CMakeLists.txt +++ b/contrib/libunwind/CMakeLists.txt @@ -1,56 +1,110 @@ - enable_language(ASM) -add_library(unwind -src/mi/init.c -src/mi/flush_cache.c -src/mi/mempool.c -src/mi/strerror.c -src/x86_64/is_fpreg.c -src/x86_64/regname.c -src/mi/_ReadULEB.c -src/mi/_ReadSLEB.c -src/mi/backtrace.c -src/mi/dyn-cancel.c -src/mi/dyn-info-list.c -src/mi/dyn-register.c -src/mi/Ldyn-extract.c -src/mi/Lfind_dynamic_proc_info.c -src/mi/Lget_accessors.c -src/mi/Lget_proc_info_by_ip.c -src/mi/Lget_proc_name.c -src/mi/Lput_dynamic_unwind_info.c -src/mi/Ldestroy_addr_space.c -src/mi/Lget_reg.c -src/mi/Lset_reg.c -src/mi/Lget_fpreg.c -src/mi/Lset_fpreg.c -src/mi/Lset_caching_policy.c -src/x86_64/setcontext.S -src/x86_64/Lcreate_addr_space.c -src/x86_64/Lget_save_loc.c -src/x86_64/Lglobal.c -src/x86_64/Linit.c -src/x86_64/Linit_local.c -src/x86_64/Linit_remote.c -src/x86_64/Lget_proc_info.c -src/x86_64/Lregs.c -src/x86_64/Lresume.c -src/x86_64/Lstash_frame.c -src/x86_64/Lstep.c -src/x86_64/Ltrace.c -src/x86_64/getcontext.S -src/dwarf/Lexpr.c -src/dwarf/Lfde.c -src/dwarf/Lfind_proc_info-lsb.c -src/dwarf/Lparser.c -src/dwarf/Lpe.c -src/dwarf/global.c -src/elf64.c +if (ARCH_PPC64LE) + add_library(unwind + src/mi/init.c + src/mi/flush_cache.c + src/mi/mempool.c + src/mi/strerror.c + src/mi/_ReadULEB.c + src/mi/_ReadSLEB.c + src/mi/backtrace.c + src/mi/dyn-cancel.c + src/mi/dyn-info-list.c + src/mi/dyn-register.c + src/mi/Ldyn-extract.c + src/mi/Lfind_dynamic_proc_info.c + src/mi/Lget_accessors.c + src/mi/Lget_proc_info_by_ip.c + src/mi/Lget_proc_name.c + src/mi/Lput_dynamic_unwind_info.c + src/mi/Ldestroy_addr_space.c + src/mi/Lget_reg.c + src/mi/Lset_reg.c + src/mi/Lget_fpreg.c + src/mi/Lset_fpreg.c + src/mi/Lset_caching_policy.c + src/dwarf/Lexpr.c + src/dwarf/Lfde.c + src/dwarf/Lfind_proc_info-lsb.c + src/dwarf/Lparser.c + src/dwarf/Lpe.c + src/dwarf/global.c + src/elf64.c + src/os-linux.c -src/os-linux.c -src/x86_64/Los-linux.c -) + src/ppc64/is_fpreg.c + src/ppc64/regname.c + src/ppc64/get_func_addr.c + src/ppc/Linit_local.c + src/ppc/Linit_remote.c + src/ppc/Lis_signal_frame.c + src/ppc/longjmp.S + src/ppc/Lreg_states_iterate.c + src/ppc/siglongjmp.S + src/ppc64/setcontext.S + src/ppc64/Lcreate_addr_space.c + src/ppc64/Lglobal.c + src/ppc64/Linit.c + src/ppc64/Lreg_states_iterate.c + src/ppc64/Lregs.c + src/ppc64/Lresume.c + src/ppc64/Lstep.c + src/ppc64/regname.c + src/ppc64/setcontext.S + ) +else () + add_library(unwind + src/mi/init.c + src/mi/flush_cache.c + src/mi/mempool.c + src/mi/strerror.c + src/mi/_ReadULEB.c + src/mi/_ReadSLEB.c + src/mi/backtrace.c + src/mi/dyn-cancel.c + src/mi/dyn-info-list.c + src/mi/dyn-register.c + src/mi/Ldyn-extract.c + src/mi/Lfind_dynamic_proc_info.c + src/mi/Lget_accessors.c + src/mi/Lget_proc_info_by_ip.c + src/mi/Lget_proc_name.c + src/mi/Lput_dynamic_unwind_info.c + src/mi/Ldestroy_addr_space.c + src/mi/Lget_reg.c + src/mi/Lset_reg.c + src/mi/Lget_fpreg.c + src/mi/Lset_fpreg.c + src/mi/Lset_caching_policy.c + src/dwarf/Lexpr.c + src/dwarf/Lfde.c + src/dwarf/Lfind_proc_info-lsb.c + src/dwarf/Lparser.c + src/dwarf/Lpe.c + src/dwarf/global.c + src/elf64.c + src/os-linux.c + + src/x86_64/is_fpreg.c + src/x86_64/regname.c + src/x86_64/setcontext.S + src/x86_64/Lcreate_addr_space.c + src/x86_64/Lget_save_loc.c + src/x86_64/Lglobal.c + src/x86_64/Linit.c + src/x86_64/Linit_local.c + src/x86_64/Linit_remote.c + src/x86_64/Lget_proc_info.c + src/x86_64/Lregs.c + src/x86_64/Lresume.c + src/x86_64/Lstash_frame.c + src/x86_64/Lstep.c + src/x86_64/Ltrace.c + src/x86_64/getcontext.S + src/x86_64/Los-linux.c + ) +endif() find_file (HAVE_ATOMIC_OPS_H "atomic_ops.h") configure_file (config/config.h.in ${CMAKE_CURRENT_BINARY_DIR}/config/config.h) diff --git a/contrib/libunwind/include/tdep-ppc64/libunwind_i.h b/contrib/libunwind/include/tdep-ppc64/libunwind_i.h index 975f3bb3662..5c50ad63942 100644 --- a/contrib/libunwind/include/tdep-ppc64/libunwind_i.h +++ b/contrib/libunwind/include/tdep-ppc64/libunwind_i.h @@ -37,6 +37,7 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. */ #include #include +#include #include "elf64.h" #include "mempool.h" diff --git a/contrib/libunwind/src/ppc/Greg_states_iterate.c b/contrib/libunwind/src/ppc/Greg_states_iterate.c index a39837a1781..99d967daebc 100644 --- a/contrib/libunwind/src/ppc/Greg_states_iterate.c +++ b/contrib/libunwind/src/ppc/Greg_states_iterate.c @@ -25,7 +25,7 @@ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. */ -#include "unwind_i.h" +#include "../ppc64/unwind_i.h" PROTECTED int unw_reg_states_iterate (unw_cursor_t *cursor, diff --git a/dbms/CMakeLists.txt b/dbms/CMakeLists.txt index 3ce29e9d65e..93d9a0a1a11 100644 --- a/dbms/CMakeLists.txt +++ b/dbms/CMakeLists.txt @@ -298,6 +298,11 @@ target_link_libraries(dbms PRIVATE ${OPENSSL_CRYPTO_LIBRARY} Threads::Threads) target_include_directories (dbms SYSTEM BEFORE PRIVATE ${DIVIDE_INCLUDE_DIR}) target_include_directories (dbms SYSTEM BEFORE PRIVATE ${SPARCEHASH_INCLUDE_DIR}) +if (USE_PROTOBUF) + target_link_libraries (dbms PRIVATE ${Protobuf_LIBRARY}) + target_include_directories (dbms SYSTEM BEFORE PRIVATE ${Protobuf_INCLUDE_DIR}) +endif () + if (USE_HDFS) target_link_libraries (clickhouse_common_io PRIVATE ${HDFS3_LIBRARY}) target_include_directories (clickhouse_common_io SYSTEM BEFORE PRIVATE ${HDFS3_INCLUDE_DIR}) @@ -318,7 +323,7 @@ target_include_directories (clickhouse_common_io BEFORE PRIVATE ${COMMON_INCLUDE add_subdirectory (programs) add_subdirectory (tests) -if (ENABLE_TESTS) +if (ENABLE_TESTS AND USE_GTEST) macro (grep_gtest_sources BASE_DIR DST_VAR) # Cold match files that are not in tests/ directories file(GLOB_RECURSE "${DST_VAR}" RELATIVE "${BASE_DIR}" "gtest*.cpp") diff --git a/dbms/cmake/version.cmake b/dbms/cmake/version.cmake index c684c9a4f7c..b39f7d14b2b 100644 --- a/dbms/cmake/version.cmake +++ b/dbms/cmake/version.cmake @@ -2,10 +2,10 @@ set(VERSION_REVISION 54413) set(VERSION_MAJOR 19) set(VERSION_MINOR 1) -set(VERSION_PATCH 5) -set(VERSION_GITHASH 2a7e7364c139b3c97f54f38ca6ea76ab4fa61e4b) -set(VERSION_DESCRIBE v19.1.5-testing) -set(VERSION_STRING 19.1.5) +set(VERSION_PATCH 6) +set(VERSION_GITHASH f73b337a93d534671b2187660398b8573fc1d464) +set(VERSION_DESCRIBE v19.1.6-testing) +set(VERSION_STRING 19.1.6) # end of autochange set(VERSION_EXTRA "" CACHE STRING "") diff --git a/dbms/programs/client/Client.cpp b/dbms/programs/client/Client.cpp index dd3fb2b84b6..65565286a98 100644 --- a/dbms/programs/client/Client.cpp +++ b/dbms/programs/client/Client.cpp @@ -56,6 +56,7 @@ #include #include #include +#include #include #include #include @@ -219,6 +220,9 @@ private: APPLY_FOR_SETTINGS(EXTRACT_SETTING) #undef EXTRACT_SETTING + /// Set path for format schema files + if (config().has("format_schema_path")) + context.setFormatSchemaPath(Poco::Path(config().getString("format_schema_path")).toString()); } @@ -1206,6 +1210,10 @@ private: const auto & id = typeid_cast(*query_with_output->format); current_format = id.name; } + if (query_with_output->settings_ast) + { + InterpreterSetQuery(query_with_output->settings_ast, context).executeForCurrentContext(); + } } if (has_vertical_output_suffix) diff --git a/dbms/programs/performance-test/PerformanceTest.cpp b/dbms/programs/performance-test/PerformanceTest.cpp index 27bf986fc1b..e91365aeade 100644 --- a/dbms/programs/performance-test/PerformanceTest.cpp +++ b/dbms/programs/performance-test/PerformanceTest.cpp @@ -46,6 +46,8 @@ namespace fs = boost::filesystem; using String = std::string; const String FOUR_SPACES = " "; +const std::regex QUOTE_REGEX{"\""}; +const std::regex NEW_LINE{"\n"}; namespace DB { @@ -80,7 +82,7 @@ public: bool reserved = (value[0] == '[' || value[0] == '{' || value == "null"); if (!reserved && wrap) - value = '"' + value + '"'; + value = '"' + std::regex_replace(value, NEW_LINE, "\\n") + '"'; content[key] = value; } @@ -579,7 +581,8 @@ private: using Paths = std::vector; using StringToVector = std::map>; - StringToVector substitutions; + using StringToMap = std::map; + StringToMap substitutions; using StringKeyValue = std::map; std::vector substitutions_maps; @@ -933,13 +936,13 @@ private: { /// Make "subconfig" of inner xml block ConfigurationPtr substitutions_view(test_config->createView("substitutions")); - constructSubstitutions(substitutions_view, substitutions); + constructSubstitutions(substitutions_view, substitutions[test_name]); auto queries_pre_format = queries; queries.clear(); for (const auto & query : queries_pre_format) { - auto formatted = formatQueries(query, substitutions); + auto formatted = formatQueries(query, substitutions[test_name]); queries.insert(queries.end(), formatted.begin(), formatted.end()); } } @@ -994,6 +997,9 @@ private: } else { + if (metrics.empty()) + throw DB::Exception("You shoud specify at least one metric", DB::ErrorCodes::BAD_ARGUMENTS); + main_metric = metrics[0]; if (lite_output) throw DB::Exception("Specify main_metric for lite output", DB::ErrorCodes::BAD_ARGUMENTS); } @@ -1219,11 +1225,11 @@ public: json_output.set("test_name", test_name); json_output.set("main_metric", main_metric); - if (substitutions.size()) + if (substitutions[test_name].size()) { JSONString json_parameters(2); /// here, 2 is the size of \t padding - for (auto it = substitutions.begin(); it != substitutions.end(); ++it) + for (auto it = substitutions[test_name].begin(); it != substitutions[test_name].end(); ++it) { String parameter = it->first; std::vector values = it->second; @@ -1231,7 +1237,7 @@ public: String array_string = "["; for (size_t i = 0; i != values.size(); ++i) { - array_string += '"' + values[i] + '"'; + array_string += '"' + std::regex_replace(values[i], QUOTE_REGEX, "\\\"") + '"'; if (i != values.size() - 1) { array_string += ", "; @@ -1257,7 +1263,7 @@ public: JSONString runJSON; - runJSON.set("query", queries[query_index]); + runJSON.set("query", std::regex_replace(queries[query_index], QUOTE_REGEX, "\\\"")); if (!statistics.exception.empty()) runJSON.set("exception", statistics.exception); diff --git a/dbms/programs/server/CMakeLists.txt b/dbms/programs/server/CMakeLists.txt index 9de696e417f..f8e805fbc74 100644 --- a/dbms/programs/server/CMakeLists.txt +++ b/dbms/programs/server/CMakeLists.txt @@ -23,7 +23,7 @@ if (CLICKHOUSE_SPLIT_BINARY) install (TARGETS clickhouse-server ${CLICKHOUSE_ALL_TARGETS} RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR} COMPONENT clickhouse) endif () -if (OS_LINUX AND MAKE_STATIC_LIBRARIES) +if (GLIBC_COMPATIBILITY) set (GLIBC_MAX_REQUIRED 2.4 CACHE INTERNAL "") # temporary disabled. to enable - change 'exit 0' to 'exit $a' add_test(NAME GLIBC_required_version COMMAND bash -c "readelf -s ${CMAKE_CURRENT_BINARY_DIR}/../clickhouse-server | perl -nE 'END {exit 0 if $a} ++$a, print if /\\x40GLIBC_(\\S+)/ and pack(q{C*}, split /\\./, \$1) gt pack q{C*}, split /\\./, q{${GLIBC_MAX_REQUIRED}}'") diff --git a/dbms/programs/server/HTTPHandler.cpp b/dbms/programs/server/HTTPHandler.cpp index d86c526784b..01968e19224 100644 --- a/dbms/programs/server/HTTPHandler.cpp +++ b/dbms/programs/server/HTTPHandler.cpp @@ -31,7 +31,7 @@ #include #include -#include +#include #include #include diff --git a/dbms/programs/server/Server.cpp b/dbms/programs/server/Server.cpp index c64b8358612..fd43c322add 100644 --- a/dbms/programs/server/Server.cpp +++ b/dbms/programs/server/Server.cpp @@ -418,7 +418,7 @@ int Server::main(const std::vector & /*args*/) /// Set path for format schema files auto format_schema_path = Poco::File(config().getString("format_schema_path", path + "format_schemas/")); - global_context->setFormatSchemaPath(format_schema_path.path() + "/"); + global_context->setFormatSchemaPath(format_schema_path.path()); format_schema_path.createDirectories(); LOG_INFO(log, "Loading metadata."); diff --git a/dbms/programs/server/TCPHandler.cpp b/dbms/programs/server/TCPHandler.cpp index c3dff11146e..fbf4e22120a 100644 --- a/dbms/programs/server/TCPHandler.cpp +++ b/dbms/programs/server/TCPHandler.cpp @@ -485,53 +485,44 @@ void TCPHandler::processTablesStatusRequest() void TCPHandler::sendProfileInfo() { - if (const IProfilingBlockInputStream * input = dynamic_cast(state.io.in.get())) - { - writeVarUInt(Protocol::Server::ProfileInfo, *out); - input->getProfileInfo().write(*out); - out->next(); - } + writeVarUInt(Protocol::Server::ProfileInfo, *out); + state.io.in->getProfileInfo().write(*out); + out->next(); } void TCPHandler::sendTotals() { - if (IProfilingBlockInputStream * input = dynamic_cast(state.io.in.get())) + const Block & totals = state.io.in->getTotals(); + + if (totals) { - const Block & totals = input->getTotals(); + initBlockOutput(totals); - if (totals) - { - initBlockOutput(totals); + writeVarUInt(Protocol::Server::Totals, *out); + writeStringBinary("", *out); - writeVarUInt(Protocol::Server::Totals, *out); - writeStringBinary("", *out); - - state.block_out->write(totals); - state.maybe_compressed_out->next(); - out->next(); - } + state.block_out->write(totals); + state.maybe_compressed_out->next(); + out->next(); } } void TCPHandler::sendExtremes() { - if (IProfilingBlockInputStream * input = dynamic_cast(state.io.in.get())) + Block extremes = state.io.in->getExtremes(); + + if (extremes) { - Block extremes = input->getExtremes(); + initBlockOutput(extremes); - if (extremes) - { - initBlockOutput(extremes); + writeVarUInt(Protocol::Server::Extremes, *out); + writeStringBinary("", *out); - writeVarUInt(Protocol::Server::Extremes, *out); - writeStringBinary("", *out); - - state.block_out->write(extremes); - state.maybe_compressed_out->next(); - out->next(); - } + state.block_out->write(extremes); + state.maybe_compressed_out->next(); + out->next(); } } diff --git a/dbms/src/AggregateFunctions/AggregateFunctionQuantile.h b/dbms/src/AggregateFunctions/AggregateFunctionQuantile.h index ad25ff95af3..cee2b6fe0c0 100644 --- a/dbms/src/AggregateFunctions/AggregateFunctionQuantile.h +++ b/dbms/src/AggregateFunctions/AggregateFunctionQuantile.h @@ -100,7 +100,7 @@ public: return res; } - void NO_SANITIZE_UNDEFINED add(AggregateDataPtr place, const IColumn ** columns, size_t row_num, Arena *) const override + void add(AggregateDataPtr place, const IColumn ** columns, size_t row_num, Arena *) const override { /// Out of range conversion may occur. This is Ok. @@ -177,8 +177,11 @@ public: static void assertSecondArg(const DataTypes & argument_types) { if constexpr (has_second_arg) - /// TODO: check that second argument is of numerical type. + { assertBinary(Name::name, argument_types); + if (!isUnsignedInteger(argument_types[1])) + throw Exception("Second argument (weight) for function " + std::string(Name::name) + " must be unsigned integer, but it has type " + argument_types[1]->getName(), ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); + } else assertUnary(Name::name, argument_types); } diff --git a/dbms/src/AggregateFunctions/AggregateFunctionSumMap.cpp b/dbms/src/AggregateFunctions/AggregateFunctionSumMap.cpp index 571d6f5c0a1..75cd62c00f1 100644 --- a/dbms/src/AggregateFunctions/AggregateFunctionSumMap.cpp +++ b/dbms/src/AggregateFunctions/AggregateFunctionSumMap.cpp @@ -12,10 +12,41 @@ namespace DB namespace { -AggregateFunctionPtr createAggregateFunctionSumMap(const std::string & name, const DataTypes & arguments, const Array & params) +struct WithOverflowPolicy { - assertNoParameters(name, params); + /// Overflow, meaning that the returned type is the same as the input type. + static DataTypePtr promoteType(const DataTypePtr & data_type) { return data_type; } +}; +struct WithoutOverflowPolicy +{ + /// No overflow, meaning we promote the types if necessary. + static DataTypePtr promoteType(const DataTypePtr & data_type) + { + if (!data_type->canBePromoted()) + throw new Exception{"Values to be summed are expected to be Numeric, Float or Decimal.", + ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT}; + + return data_type->promoteNumericType(); + } +}; + +template +using SumMapWithOverflow = AggregateFunctionSumMap; + +template +using SumMapWithoutOverflow = AggregateFunctionSumMap; + +template +using SumMapFilteredWithOverflow = AggregateFunctionSumMapFiltered; + +template +using SumMapFilteredWithoutOverflow = AggregateFunctionSumMapFiltered; + +using SumMapArgs = std::pair; + +SumMapArgs parseArguments(const std::string & name, const DataTypes & arguments) +{ if (arguments.size() < 2) throw Exception("Aggregate function " + name + " requires at least two arguments of Array type.", ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH); @@ -25,9 +56,11 @@ AggregateFunctionPtr createAggregateFunctionSumMap(const std::string & name, con throw Exception("First argument for function " + name + " must be an array.", ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT); - const DataTypePtr & keys_type = array_type->getNestedType(); + + DataTypePtr keys_type = array_type->getNestedType(); DataTypes values_types; + values_types.reserve(arguments.size() - 1); for (size_t i = 1; i < arguments.size(); ++i) { array_type = checkAndGetDataType(arguments[i].get()); @@ -37,20 +70,55 @@ AggregateFunctionPtr createAggregateFunctionSumMap(const std::string & name, con values_types.push_back(array_type->getNestedType()); } - AggregateFunctionPtr res(createWithNumericBasedType(*keys_type, keys_type, values_types)); + return {std::move(keys_type), std::move(values_types)}; +} + +template