mirror of
https://github.com/ClickHouse/ClickHouse.git
synced 2024-11-22 07:31:57 +00:00
Merge branch 'master' into libcxx-as-submodule
This commit is contained in:
Alexey Milovidov
commit
880cbbb632
30
.github/ISSUE_TEMPLATE/bug_report.md
vendored
Normal file
30
.github/ISSUE_TEMPLATE/bug_report.md
vendored
Normal file
@ -0,0 +1,30 @@
|
|||||||
|
---
|
||||||
|
name: Bug report
|
||||||
|
about: Create a report to help us improve ClickHouse
|
||||||
|
title: ''
|
||||||
|
labels: bug, issue
|
||||||
|
assignees: ''
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
(you don't have to strictly follow this form)
|
||||||
|
|
||||||
|
**Describe the bug**
|
||||||
|
A clear and concise description of what the bug is.
|
||||||
|
|
||||||
|
**How to reproduce**
|
||||||
|
* Which ClickHouse server version to use
|
||||||
|
* Which interface to use, if matters
|
||||||
|
* Non-default settings, if any
|
||||||
|
* `CREATE TABLE` statements for all tables involved
|
||||||
|
* Sample data for all these tables, use [clickhouse-obfuscator](https://github.com/yandex/ClickHouse/blob/master/dbms/programs/obfuscator/Obfuscator.cpp#L42-L80) if necessary
|
||||||
|
* Queries to run that lead to unexpected result
|
||||||
|
|
||||||
|
**Expected behavior**
|
||||||
|
A clear and concise description of what you expected to happen.
|
||||||
|
|
||||||
|
**Error message and/or stacktrace**
|
||||||
|
If applicable, add screenshots to help explain your problem.
|
||||||
|
|
||||||
|
**Additional context**
|
||||||
|
Add any other context about the problem here.
|
||||||
21
.github/ISSUE_TEMPLATE/build-issue.md
vendored
Normal file
21
.github/ISSUE_TEMPLATE/build-issue.md
vendored
Normal file
@ -0,0 +1,21 @@
|
|||||||
|
---
|
||||||
|
name: Build issue
|
||||||
|
about: Report failed ClickHouse build from master
|
||||||
|
title: ''
|
||||||
|
labels: build
|
||||||
|
assignees: ''
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
Make sure that `git diff` result is empty and you've just pulled fresh master. Try cleaning up cmake cache. Just in case, official build instructions are published here: https://clickhouse.yandex/docs/en/development/build/
|
||||||
|
|
||||||
|
**Operating system**
|
||||||
|
OS kind or distribution, specific version/release, non-standard kernel if any. If you are trying to build inside virtual machine, please mention it too.
|
||||||
|
|
||||||
|
**Cmake version**
|
||||||
|
|
||||||
|
**Ninja version**
|
||||||
|
|
||||||
|
**Compiler name and version**
|
||||||
|
|
||||||
|
**Full cmake and/or ninja output**
|
||||||
22
.github/ISSUE_TEMPLATE/feature_request.md
vendored
Normal file
22
.github/ISSUE_TEMPLATE/feature_request.md
vendored
Normal file
@ -0,0 +1,22 @@
|
|||||||
|
---
|
||||||
|
name: Feature request
|
||||||
|
about: Suggest an idea for ClickHouse
|
||||||
|
title: ''
|
||||||
|
labels: feature
|
||||||
|
assignees: ''
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
(you don't have to strictly follow this form)
|
||||||
|
|
||||||
|
**Use case**
|
||||||
|
A clear and concise description of what is the intended usage scenario is.
|
||||||
|
|
||||||
|
**Describe the solution you'd like**
|
||||||
|
A clear and concise description of what you want to happen.
|
||||||
|
|
||||||
|
**Describe alternatives you've considered**
|
||||||
|
A clear and concise description of any alternative solutions or features you've considered.
|
||||||
|
|
||||||
|
**Additional context**
|
||||||
|
Add any other context or screenshots about the feature request here.
|
||||||
12
.github/ISSUE_TEMPLATE/question.md
vendored
Normal file
12
.github/ISSUE_TEMPLATE/question.md
vendored
Normal file
@ -0,0 +1,12 @@
|
|||||||
|
---
|
||||||
|
name: Question
|
||||||
|
about: Ask question about ClickHouse
|
||||||
|
title: ''
|
||||||
|
labels: question
|
||||||
|
assignees: ''
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
Make sure to check documentation https://clickhouse.yandex/docs/en/ first. If the question is concise and probably has a short answer, asking it in Telegram chat https://telegram.me/clickhouse_en is probably the fastest way to find the answer. For more complicated questions, consider asking them on StackOverflow with "clickhouse" tag https://stackoverflow.com/questions/tagged/clickhouse
|
||||||
|
|
||||||
|
If you still prefer GitHub issues, remove all this text and ask your question here.
|
||||||
16
.gitignore
vendored
16
.gitignore
vendored
@ -180,18 +180,10 @@ utils/zookeeper-create-entry-to-download-part/zookeeper-create-entry-to-download
|
|||||||
utils/zookeeper-dump-tree/zookeeper-dump-tree
|
utils/zookeeper-dump-tree/zookeeper-dump-tree
|
||||||
utils/zookeeper-remove-by-list/zookeeper-remove-by-list
|
utils/zookeeper-remove-by-list/zookeeper-remove-by-list
|
||||||
dbms/src/Storages/tests/remove_symlink_directory
|
dbms/src/Storages/tests/remove_symlink_directory
|
||||||
debian/control
|
|
||||||
debian/copyright
|
|
||||||
debian/tmp/
|
|
||||||
libs/libcommon/src/tests/json_test
|
libs/libcommon/src/tests/json_test
|
||||||
utils/compressor/zstd_test
|
utils/compressor/zstd_test
|
||||||
utils/wikistat-loader/wikistat-loader
|
utils/wikistat-loader/wikistat-loader
|
||||||
dbms/src/Common/tests/pod_array
|
dbms/src/Common/tests/pod_array
|
||||||
debian/clickhouse-benchmark/
|
|
||||||
debian/clickhouse-client/
|
|
||||||
debian/clickhouse-server-base/
|
|
||||||
debian/clickhouse-server-common/
|
|
||||||
debian/files
|
|
||||||
|
|
||||||
dbms/src/Server/data/*
|
dbms/src/Server/data/*
|
||||||
dbms/src/Server/metadata/*
|
dbms/src/Server/metadata/*
|
||||||
@ -210,9 +202,6 @@ vgcore*
|
|||||||
*.changes
|
*.changes
|
||||||
build-stamp
|
build-stamp
|
||||||
configure-stamp
|
configure-stamp
|
||||||
debian/*.debhelper.log
|
|
||||||
debian/*.debhelper
|
|
||||||
debian/*.substvars
|
|
||||||
|
|
||||||
*.bin
|
*.bin
|
||||||
*.mrk
|
*.mrk
|
||||||
@ -251,3 +240,8 @@ website/package-lock.json
|
|||||||
|
|
||||||
# cquery cache
|
# cquery cache
|
||||||
/.cquery-cache
|
/.cquery-cache
|
||||||
|
|
||||||
|
# ccls cache
|
||||||
|
/.ccls-cache
|
||||||
|
|
||||||
|
/compile_commands.json
|
||||||
|
|||||||
17
.gitmodules
vendored
17
.gitmodules
vendored
@ -48,10 +48,16 @@
|
|||||||
url = https://github.com/ClickHouse-Extras/protobuf.git
|
url = https://github.com/ClickHouse-Extras/protobuf.git
|
||||||
[submodule "contrib/boost"]
|
[submodule "contrib/boost"]
|
||||||
path = contrib/boost
|
path = contrib/boost
|
||||||
url = https://github.com/ClickHouse-Extras/boost-extra.git
|
url = https://github.com/ClickHouse-Extras/boost.git
|
||||||
[submodule "contrib/base64"]
|
[submodule "contrib/base64"]
|
||||||
path = contrib/base64
|
path = contrib/base64
|
||||||
url = https://github.com/aklomp/base64.git
|
url = https://github.com/aklomp/base64.git
|
||||||
|
[submodule "contrib/arrow"]
|
||||||
|
path = contrib/arrow
|
||||||
|
url = https://github.com/apache/arrow
|
||||||
|
[submodule "contrib/thrift"]
|
||||||
|
path = contrib/thrift
|
||||||
|
url = https://github.com/apache/thrift.git
|
||||||
[submodule "contrib/libhdfs3"]
|
[submodule "contrib/libhdfs3"]
|
||||||
path = contrib/libhdfs3
|
path = contrib/libhdfs3
|
||||||
url = https://github.com/ClickHouse-Extras/libhdfs3.git
|
url = https://github.com/ClickHouse-Extras/libhdfs3.git
|
||||||
@ -67,3 +73,12 @@
|
|||||||
[submodule "contrib/libcxxabi"]
|
[submodule "contrib/libcxxabi"]
|
||||||
path = contrib/libcxxabi
|
path = contrib/libcxxabi
|
||||||
url = https://github.com/llvm-mirror/libcxxabi.git
|
url = https://github.com/llvm-mirror/libcxxabi.git
|
||||||
|
[submodule "contrib/snappy"]
|
||||||
|
path = contrib/snappy
|
||||||
|
url = https://github.com/google/snappy
|
||||||
|
[submodule "contrib/cppkafka"]
|
||||||
|
path = contrib/cppkafka
|
||||||
|
url = https://github.com/ClickHouse-Extras/cppkafka.git
|
||||||
|
[submodule "contrib/brotli"]
|
||||||
|
path = contrib/brotli
|
||||||
|
url = https://github.com/google/brotli.git
|
||||||
|
|||||||
@ -1,50 +0,0 @@
|
|||||||
language: generic
|
|
||||||
|
|
||||||
matrix:
|
|
||||||
fast_finish: true
|
|
||||||
include:
|
|
||||||
# We need to have gcc7 headers to compile c++17 code on clang
|
|
||||||
# - os: linux
|
|
||||||
#
|
|
||||||
# cache:
|
|
||||||
# ccache: true
|
|
||||||
# timeout: 1000
|
|
||||||
# directories:
|
|
||||||
# - /home/travis/.ccache
|
|
||||||
#
|
|
||||||
# addons:
|
|
||||||
# apt:
|
|
||||||
# update: true
|
|
||||||
# sources:
|
|
||||||
# - ubuntu-toolchain-r-test
|
|
||||||
# - llvm-toolchain-trusty-5.0
|
|
||||||
# packages: [ ninja-build, g++-7, clang-5.0, lld-5.0, libicu-dev, libreadline-dev, libmysqlclient-dev, unixodbc-dev, libltdl-dev, libssl-dev, libboost-dev, zlib1g-dev, libdouble-conversion-dev, libsparsehash-dev, librdkafka-dev, libcapnp-dev, libsparsehash-dev, libgoogle-perftools-dev, bash, expect, python, python-lxml, python-termcolor, curl, perl, sudo, openssl]
|
|
||||||
#
|
|
||||||
# env:
|
|
||||||
# - MATRIX_EVAL="export CC=clang-5.0 CXX=clang++-5.0"
|
|
||||||
#
|
|
||||||
# script:
|
|
||||||
# - utils/travis/normal.sh
|
|
||||||
|
|
||||||
- os: linux
|
|
||||||
|
|
||||||
sudo: required
|
|
||||||
|
|
||||||
cache:
|
|
||||||
timeout: 1000
|
|
||||||
directories:
|
|
||||||
- /var/cache/pbuilder/ccache
|
|
||||||
|
|
||||||
addons:
|
|
||||||
apt:
|
|
||||||
update: true
|
|
||||||
packages: [ pbuilder, fakeroot, debhelper ]
|
|
||||||
|
|
||||||
script:
|
|
||||||
- utils/travis/pbuilder.sh
|
|
||||||
|
|
||||||
allow_failures:
|
|
||||||
- os: osx
|
|
||||||
|
|
||||||
before_script:
|
|
||||||
- eval "${MATRIX_EVAL}"
|
|
||||||
@ -1 +0,0 @@
|
|||||||
|
|
||||||
276
CHANGELOG.md
276
CHANGELOG.md
@ -1,3 +1,279 @@
|
|||||||
|
## ClickHouse release 19.3.5, 2019-02-21
|
||||||
|
|
||||||
|
### Bug fixes
|
||||||
|
* Fixed bug with large http insert queries processing. [#4454](https://github.com/yandex/ClickHouse/pull/4454) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Fixed backward incompatibility with old versions due to wrong implementation of `send_logs_level` setting. [#4445](https://github.com/yandex/ClickHouse/pull/4445) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Fixed backward incompatibility of table function `remote` introduced with column comments. [#4446](https://github.com/yandex/ClickHouse/pull/4446) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
|
||||||
|
## ClickHouse release 19.3.4, 2019-02-16
|
||||||
|
|
||||||
|
### Improvements
|
||||||
|
* Table index size is not accounted for memory limits when doing `ATTACH TABLE` query. Avoided the possibility that a table cannot be attached after being detached. [#4396](https://github.com/yandex/ClickHouse/pull/4396) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Slightly raised up the limit on max string and array size received from ZooKeeper. It allows to continue to work with increased size of `CLIENT_JVMFLAGS=-Djute.maxbuffer=...` on ZooKeeper. [#4398](https://github.com/yandex/ClickHouse/pull/4398) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Allow to repair abandoned replica even if it already has huge number of nodes in its queue. [#4399](https://github.com/yandex/ClickHouse/pull/4399) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Add one required argument to `SET` index (max stored rows number). [#4386](https://github.com/yandex/ClickHouse/pull/4386) ([Nikita Vasilev](https://github.com/nikvas0))
|
||||||
|
|
||||||
|
### Bug Fixes
|
||||||
|
* Fixed `WITH ROLLUP` result for group by single `LowCardinality` key. [#4384](https://github.com/yandex/ClickHouse/pull/4384) ([Nikolai Kochetov](https://github.com/KochetovNicolai))
|
||||||
|
* Fixed bug in the set index (dropping a granule if it contains more than `max_rows` rows). [#4386](https://github.com/yandex/ClickHouse/pull/4386) ([Nikita Vasilev](https://github.com/nikvas0))
|
||||||
|
* A lot of FreeBSD build fixes. [#4397](https://github.com/yandex/ClickHouse/pull/4397) ([proller](https://github.com/proller))
|
||||||
|
* Fixed aliases substitution in queries with subquery containing same alias (issue [#4110](https://github.com/yandex/ClickHouse/issues/4110)). [#4351](https://github.com/yandex/ClickHouse/pull/4351) ([Artem Zuikov](https://github.com/4ertus2))
|
||||||
|
|
||||||
|
### Build/Testing/Packaging Improvements
|
||||||
|
* Add ability to run `clickhouse-server` for stateless tests in docker image. [#4347](https://github.com/yandex/ClickHouse/pull/4347) ([Vasily Nemkov](https://github.com/Enmk))
|
||||||
|
|
||||||
|
## ClickHouse release 19.3.3, 2019-02-13
|
||||||
|
|
||||||
|
### New Features
|
||||||
|
* Added the `KILL MUTATION` statement that allows removing mutations that are for some reasons stuck. Added `latest_failed_part`, `latest_fail_time`, `latest_fail_reason` fields to the `system.mutations` table for easier troubleshooting. [#4287](https://github.com/yandex/ClickHouse/pull/4287) ([Alex Zatelepin](https://github.com/ztlpn))
|
||||||
|
* Added aggregate function `entropy` which computes Shannon entropy. [#4238](https://github.com/yandex/ClickHouse/pull/4238) ([Quid37](https://github.com/Quid37))
|
||||||
|
* Added ability to send queries `INSERT INTO tbl VALUES (....` to server without splitting on `query` and `data` parts. [#4301](https://github.com/yandex/ClickHouse/pull/4301) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Generic implementation of `arrayWithConstant` function was added. [#4322](https://github.com/yandex/ClickHouse/pull/4322) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Implented `NOT BETWEEN` comparison operator. [#4228](https://github.com/yandex/ClickHouse/pull/4228) ([Dmitry Naumov](https://github.com/nezed))
|
||||||
|
* Implement `sumMapFiltered` in order to be able to limit the number of keys for which values will be summed by `sumMap`. [#4129](https://github.com/yandex/ClickHouse/pull/4129) ([Léo Ercolanelli](https://github.com/ercolanelli-leo))
|
||||||
|
* Added support of `Nullable` types in `mysql` table function. [#4198](https://github.com/yandex/ClickHouse/pull/4198) ([Emmanuel Donin de Rosière](https://github.com/edonin))
|
||||||
|
* Support for arbitrary constant expressions in `LIMIT` clause. [#4246](https://github.com/yandex/ClickHouse/pull/4246) ([k3box](https://github.com/k3box))
|
||||||
|
* Added `topKWeighted` aggregate function that takes additional argument with (unsigned integer) weight. [#4245](https://github.com/yandex/ClickHouse/pull/4245) ([Andrew Golman](https://github.com/andrewgolman))
|
||||||
|
* `StorageJoin` now supports `join_overwrite` setting that allows overwriting existing values of the same key. [#3973](https://github.com/yandex/ClickHouse/pull/3973) ([Amos Bird](https://github.com/amosbird)
|
||||||
|
* Added function `toStartOfInterval`. [#4304](https://github.com/yandex/ClickHouse/pull/4304) ([Vitaly Baranov](https://github.com/vitlibar))
|
||||||
|
* Added `RowBinaryWithNamesAndTypes` format. [#4200](https://github.com/yandex/ClickHouse/pull/4200) ([Oleg V. Kozlyuk](https://github.com/DarkWanderer))
|
||||||
|
* Added `IPv4` and `IPv6` data types. More effective implementations of `IPv*` functions. [#3669](https://github.com/yandex/ClickHouse/pull/3669) ([Vasily Nemkov](https://github.com/Enmk))
|
||||||
|
* Added function `toStartOfTenMinutes()`. [#4298](https://github.com/yandex/ClickHouse/pull/4298) ([Vitaly Baranov](https://github.com/vitlibar))
|
||||||
|
* Added `Protobuf` output format. [#4005](https://github.com/yandex/ClickHouse/pull/4005) [#4158](https://github.com/yandex/ClickHouse/pull/4158) ([Vitaly Baranov](https://github.com/vitlibar))
|
||||||
|
* Added brotli support for HTTP interface for data import (INSERTs). [#4235](https://github.com/yandex/ClickHouse/pull/4235) ([Mikhail ](https://github.com/fandyushin))
|
||||||
|
* Added hints while user make typo in function name or type in command line client. [#4239](https://github.com/yandex/ClickHouse/pull/4239) ([Danila Kutenin](https://github.com/danlark1))
|
||||||
|
* Added `Query-Id` to Server's HTTP Response header. [#4231](https://github.com/yandex/ClickHouse/pull/4231) ([Mikhail ](https://github.com/fandyushin))
|
||||||
|
|
||||||
|
### Experimental features
|
||||||
|
* Added `minmax` and `set` data skipping indices for MergeTree table engines family. [#4143](https://github.com/yandex/ClickHouse/pull/4143) ([Nikita Vasilev](https://github.com/nikvas0))
|
||||||
|
* Added conversion of `CROSS JOIN` to `INNER JOIN` if possible. [#4221](https://github.com/yandex/ClickHouse/pull/4221) [#4266](https://github.com/yandex/ClickHouse/pull/4266) ([Artem Zuikov](https://github.com/4ertus2))
|
||||||
|
|
||||||
|
### Bug Fixes
|
||||||
|
* Fixed `Not found column` for duplicate columns in `JOIN ON` section. [#4279](https://github.com/yandex/ClickHouse/pull/4279) ([Artem Zuikov](https://github.com/4ertus2))
|
||||||
|
* Make `START REPLICATED SENDS` command start replicated sends. [#4229](https://github.com/yandex/ClickHouse/pull/4229) ([nvartolomei](https://github.com/nvartolomei))
|
||||||
|
* Fixed aggregate functions execution with `Array(LowCardinality)` arguments. [#4055](https://github.com/yandex/ClickHouse/pull/4055) ([KochetovNicolai](https://github.com/KochetovNicolai))
|
||||||
|
* Fixed wrong behaviour when doing `INSERT ... SELECT ... FROM file(...)` query and file has `CSVWithNames` or `TSVWIthNames` format and the first data row is missing. [#4297](https://github.com/yandex/ClickHouse/pull/4297) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Fixed crash on dictionary reload if dictionary not available. This bug was appeared in 19.1.6. [#4188](https://github.com/yandex/ClickHouse/pull/4188) ([proller](https://github.com/proller))
|
||||||
|
* Fixed `ALL JOIN` with duplicates in right table. [#4184](https://github.com/yandex/ClickHouse/pull/4184) ([Artem Zuikov](https://github.com/4ertus2))
|
||||||
|
* Fixed segmentation fault with `use_uncompressed_cache=1` and exception with wrong uncompressed size. This bug was appeared in 19.1.6. [#4186](https://github.com/yandex/ClickHouse/pull/4186) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Fixed `compile_expressions` bug with comparison of big (more than int16) dates. [#4341](https://github.com/yandex/ClickHouse/pull/4341) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Fixed infinite loop when selecting from table function `numbers(0)`. [#4280](https://github.com/yandex/ClickHouse/pull/4280) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Temporarily disable predicate optimization for `ORDER BY`. [#3890](https://github.com/yandex/ClickHouse/pull/3890) ([Winter Zhang](https://github.com/zhang2014))
|
||||||
|
* Fixed `Illegal instruction` error when using base64 functions on old CPUs. This error has been reproduced only when ClickHouse was compiled with gcc-8. [#4275](https://github.com/yandex/ClickHouse/pull/4275) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Fixed `No message received` error when interacting with PostgreSQL ODBC Driver through TLS connection. Also fixes segfault when using MySQL ODBC Driver. [#4170](https://github.com/yandex/ClickHouse/pull/4170) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Fixed incorrect result when `Date` and `DateTime` arguments are used in branches of conditional operator (function `if`). Added generic case for function `if`. [#4243](https://github.com/yandex/ClickHouse/pull/4243) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* ClickHouse dictionaries now load within `clickhouse` process. [#4166](https://github.com/yandex/ClickHouse/pull/4166) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Fixed deadlock when `SELECT` from a table with `File` engine was retried after `No such file or directory` error. [#4161](https://github.com/yandex/ClickHouse/pull/4161) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Fixed race condition when selecting from `system.tables` may give `table doesn't exist` error. [#4313](https://github.com/yandex/ClickHouse/pull/4313) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* `clickhouse-client` can segfault on exit while loading data for command line suggestions if it was run in interactive mode. [#4317](https://github.com/yandex/ClickHouse/pull/4317) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Fixed a bug when the execution of mutations containing `IN` operators was producing incorrect results. [#4099](https://github.com/yandex/ClickHouse/pull/4099) ([Alex Zatelepin](https://github.com/ztlpn))
|
||||||
|
* Fixed error: if there is a database with `Dictionary` engine, all dictionaries forced to load at server startup, and if there is a dictionary with ClickHouse source from localhost, the dictionary cannot load. [#4255](https://github.com/yandex/ClickHouse/pull/4255) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Fixed error when system logs are tried to create again at server shutdown. [#4254](https://github.com/yandex/ClickHouse/pull/4254) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Correctly return the right type and properly handle locks in `joinGet` function. [#4153](https://github.com/yandex/ClickHouse/pull/4153) ([Amos Bird](https://github.com/amosbird))
|
||||||
|
* Added `sumMapWithOverflow` function. [#4151](https://github.com/yandex/ClickHouse/pull/4151) ([Léo Ercolanelli](https://github.com/ercolanelli-leo))
|
||||||
|
* Fixed segfault with `allow_experimental_multiple_joins_emulation`. [52de2c](https://github.com/yandex/ClickHouse/commit/52de2cd927f7b5257dd67e175f0a5560a48840d0) ([Artem Zuikov](https://github.com/4ertus2))
|
||||||
|
* Fixed bug with incorrect `Date` and `DateTime` comparison. [#4237](https://github.com/yandex/ClickHouse/pull/4237) ([valexey](https://github.com/valexey))
|
||||||
|
* Fixed fuzz test under undefined behavior sanitizer: added parameter type check for `quantile*Weighted` family of functions. [#4145](https://github.com/yandex/ClickHouse/pull/4145) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Fixed rare race condition when removing of old data parts can fail with `File not found` error. [#4378](https://github.com/yandex/ClickHouse/pull/4378) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Fix install package with missing /etc/clickhouse-server/config.xml. [#4343](https://github.com/yandex/ClickHouse/pull/4343) ([proller](https://github.com/proller))
|
||||||
|
|
||||||
|
|
||||||
|
### Build/Testing/Packaging Improvements
|
||||||
|
* Debian package: correct /etc/clickhouse-server/preprocessed link according to config. [#4205](https://github.com/yandex/ClickHouse/pull/4205) ([proller](https://github.com/proller))
|
||||||
|
* Various build fixes for FreeBSD. [#4225](https://github.com/yandex/ClickHouse/pull/4225) ([proller](https://github.com/proller))
|
||||||
|
* Added ability to create, fill and drop tables in perftest. [#4220](https://github.com/yandex/ClickHouse/pull/4220) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Added a script to check for duplicate includes. [#4326](https://github.com/yandex/ClickHouse/pull/4326) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Added ability to run queries by index in performance test. [#4264](https://github.com/yandex/ClickHouse/pull/4264) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Package with debug symbols is suggested to be installed. [#4274](https://github.com/yandex/ClickHouse/pull/4274) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Refactoring of performance-test. Better logging and signals handling. [#4171](https://github.com/yandex/ClickHouse/pull/4171) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Added docs to anonymized Yandex.Metrika datasets. [#4164](https://github.com/yandex/ClickHouse/pull/4164) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Аdded tool for converting an old month-partitioned part to the custom-partitioned format. [#4195](https://github.com/yandex/ClickHouse/pull/4195) ([Alex Zatelepin](https://github.com/ztlpn))
|
||||||
|
* Added docs about two datasets in s3. [#4144](https://github.com/yandex/ClickHouse/pull/4144) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Added script which creates changelog from pull requests description. [#4169](https://github.com/yandex/ClickHouse/pull/4169) [#4173](https://github.com/yandex/ClickHouse/pull/4173) ([KochetovNicolai](https://github.com/KochetovNicolai)) ([KochetovNicolai](https://github.com/KochetovNicolai))
|
||||||
|
* Added puppet module for Clickhouse. [#4182](https://github.com/yandex/ClickHouse/pull/4182) ([Maxim Fedotov](https://github.com/MaxFedotov))
|
||||||
|
* Added docs for a group of undocumented functions. [#4168](https://github.com/yandex/ClickHouse/pull/4168) ([Winter Zhang](https://github.com/zhang2014))
|
||||||
|
* ARM build fixes. [#4210](https://github.com/yandex/ClickHouse/pull/4210)[#4306](https://github.com/yandex/ClickHouse/pull/4306) [#4291](https://github.com/yandex/ClickHouse/pull/4291) ([proller](https://github.com/proller)) ([proller](https://github.com/proller))
|
||||||
|
* Dictionary tests now able to run from `ctest`. [#4189](https://github.com/yandex/ClickHouse/pull/4189) ([proller](https://github.com/proller))
|
||||||
|
* Now `/etc/ssl` is used as default directory with SSL certificates. [#4167](https://github.com/yandex/ClickHouse/pull/4167) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Added checking SSE and AVX instruction at start. [#4234](https://github.com/yandex/ClickHouse/pull/4234) ([Igr](https://github.com/igron99))
|
||||||
|
* Init script will wait server until start. [#4281](https://github.com/yandex/ClickHouse/pull/4281) ([proller](https://github.com/proller))
|
||||||
|
|
||||||
|
### Backward Incompatible Changes
|
||||||
|
* Removed `allow_experimental_low_cardinality_type` setting. `LowCardinality` data types are production ready. [#4323](https://github.com/yandex/ClickHouse/pull/4323) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Reduce mark cache size and uncompressed cache size accordingly to available memory amount. [#4240](https://github.com/yandex/ClickHouse/pull/4240) ([Lopatin Konstantin](https://github.com/k-lopatin)
|
||||||
|
* Added keyword `INDEX` in `CREATE TABLE` query. A column with name `index` must be quoted with backticks or double quotes: `` `index` ``. [#4143](https://github.com/yandex/ClickHouse/pull/4143) ([Nikita Vasilev](https://github.com/nikvas0))
|
||||||
|
* `sumMap` now promote result type instead of overflow. The old `sumMap` behavior can be obtained by using `sumMapWithOverflow` function. [#4151](https://github.com/yandex/ClickHouse/pull/4151) ([Léo Ercolanelli](https://github.com/ercolanelli-leo))
|
||||||
|
|
||||||
|
### Performance Impovements
|
||||||
|
* `std::sort` replaced by `pdqsort` for queries without `LIMIT`. [#4236](https://github.com/yandex/ClickHouse/pull/4236) ([Evgenii Pravda](https://github.com/kvinty))
|
||||||
|
* Now server reuse threads from global thread pool. This affects performance in some corner cases. [#4150](https://github.com/yandex/ClickHouse/pull/4150) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
|
||||||
|
### Improvements
|
||||||
|
* Implemented AIO support for FreeBSD. [#4305](https://github.com/yandex/ClickHouse/pull/4305) ([urgordeadbeef](https://github.com/urgordeadbeef))
|
||||||
|
* `SELECT * FROM a JOIN b USING a, b` now return `a` and `b` columns only from the left table. [#4141](https://github.com/yandex/ClickHouse/pull/4141) ([Artem Zuikov](https://github.com/4ertus2))
|
||||||
|
* Allow `-C` option of client to work as `-c` option. [#4232](https://github.com/yandex/ClickHouse/pull/4232) ([syominsergey](https://github.com/syominsergey))
|
||||||
|
* Now option `--password` used without value requires password from stdin. [#4230](https://github.com/yandex/ClickHouse/pull/4230) ([BSD_Conqueror](https://github.com/bsd-conqueror))
|
||||||
|
* Added highlighting of unescaped metacharacters in string literals that contain `LIKE` expressions or regexps. [#4327](https://github.com/yandex/ClickHouse/pull/4327) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Added cancelling of HTTP read only queries if client socket goes away. [#4213](https://github.com/yandex/ClickHouse/pull/4213) ([nvartolomei](https://github.com/nvartolomei))
|
||||||
|
* Now server reports progress to keep client connections alive. [#4215](https://github.com/yandex/ClickHouse/pull/4215) ([Ivan](https://github.com/abyss7))
|
||||||
|
* Slightly better message with reason for OPTIMIZE query with `optimize_throw_if_noop` setting enabled. [#4294](https://github.com/yandex/ClickHouse/pull/4294) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Added support of `--version` option for clickhouse server. [#4251](https://github.com/yandex/ClickHouse/pull/4251) ([Lopatin Konstantin](https://github.com/k-lopatin))
|
||||||
|
* Added `--help/-h` option to `clickhouse-server`. [#4233](https://github.com/yandex/ClickHouse/pull/4233) ([Yuriy Baranov](https://github.com/yurriy))
|
||||||
|
* Added support for scalar subqueries with aggregate function state result. [#4348](https://github.com/yandex/ClickHouse/pull/4348) ([Nikolai Kochetov](https://github.com/KochetovNicolai))
|
||||||
|
* Improved server shutdown time and ALTERs waiting time. [#4372](https://github.com/yandex/ClickHouse/pull/4372) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Added info about the replicated_can_become_leader setting to system.replicas and add logging if the replica won't try to become leader. [#4379](https://github.com/yandex/ClickHouse/pull/4379) ([Alex Zatelepin](https://github.com/ztlpn))
|
||||||
|
|
||||||
|
## ClickHouse release 19.1.9, 2019-02-21
|
||||||
|
|
||||||
|
### Bug fixes
|
||||||
|
* Fixed backward incompatibility with old versions due to wrong implementation of `send_logs_level` setting. [#4445](https://github.com/yandex/ClickHouse/pull/4445) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Fixed backward incompatibility of table function `remote` introduced with column comments. [#4446](https://github.com/yandex/ClickHouse/pull/4446) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
|
||||||
|
## ClickHouse release 19.1.8, 2019-02-16
|
||||||
|
|
||||||
|
### Bug Fixes
|
||||||
|
* Fix install package with missing /etc/clickhouse-server/config.xml. [#4343](https://github.com/yandex/ClickHouse/pull/4343) ([proller](https://github.com/proller))
|
||||||
|
|
||||||
|
|
||||||
|
## ClickHouse release 19.1.7, 2019-02-15
|
||||||
|
|
||||||
|
### Bug Fixes
|
||||||
|
* Correctly return the right type and properly handle locks in `joinGet` function. [#4153](https://github.com/yandex/ClickHouse/pull/4153) ([Amos Bird](https://github.com/amosbird))
|
||||||
|
* Fixed error when system logs are tried to create again at server shutdown. [#4254](https://github.com/yandex/ClickHouse/pull/4254) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Fixed error: if there is a database with `Dictionary` engine, all dictionaries forced to load at server startup, and if there is a dictionary with ClickHouse source from localhost, the dictionary cannot load. [#4255](https://github.com/yandex/ClickHouse/pull/4255) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Fixed a bug when the execution of mutations containing `IN` operators was producing incorrect results. [#4099](https://github.com/yandex/ClickHouse/pull/4099) ([Alex Zatelepin](https://github.com/ztlpn))
|
||||||
|
* `clickhouse-client` can segfault on exit while loading data for command line suggestions if it was run in interactive mode. [#4317](https://github.com/yandex/ClickHouse/pull/4317) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Fixed race condition when selecting from `system.tables` may give `table doesn't exist` error. [#4313](https://github.com/yandex/ClickHouse/pull/4313) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Fixed deadlock when `SELECT` from a table with `File` engine was retried after `No such file or directory` error. [#4161](https://github.com/yandex/ClickHouse/pull/4161) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Fixed an issue: local ClickHouse dictionaries are loaded via TCP, but should load within process. [#4166](https://github.com/yandex/ClickHouse/pull/4166) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Fixed `No message received` error when interacting with PostgreSQL ODBC Driver through TLS connection. Also fixes segfault when using MySQL ODBC Driver. [#4170](https://github.com/yandex/ClickHouse/pull/4170) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Temporarily disable predicate optimization for `ORDER BY`. [#3890](https://github.com/yandex/ClickHouse/pull/3890) ([Winter Zhang](https://github.com/zhang2014))
|
||||||
|
* Fixed infinite loop when selecting from table function `numbers(0)`. [#4280](https://github.com/yandex/ClickHouse/pull/4280) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Fixed `compile_expressions` bug with comparison of big (more than int16) dates. [#4341](https://github.com/yandex/ClickHouse/pull/4341) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Fixed segmentation fault with `uncompressed_cache=1` and exception with wrong uncompressed size. [#4186](https://github.com/yandex/ClickHouse/pull/4186) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Fixed `ALL JOIN` with duplicates in right table. [#4184](https://github.com/yandex/ClickHouse/pull/4184) ([Artem Zuikov](https://github.com/4ertus2))
|
||||||
|
* Fixed wrong behaviour when doing `INSERT ... SELECT ... FROM file(...)` query and file has `CSVWithNames` or `TSVWIthNames` format and the first data row is missing. [#4297](https://github.com/yandex/ClickHouse/pull/4297) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Fixed aggregate functions execution with `Array(LowCardinality)` arguments. [#4055](https://github.com/yandex/ClickHouse/pull/4055) ([KochetovNicolai](https://github.com/KochetovNicolai))
|
||||||
|
* Debian package: correct /etc/clickhouse-server/preprocessed link according to config. [#4205](https://github.com/yandex/ClickHouse/pull/4205) ([proller](https://github.com/proller))
|
||||||
|
* Fixed fuzz test under undefined behavior sanitizer: added parameter type check for `quantile*Weighted` family of functions. [#4145](https://github.com/yandex/ClickHouse/pull/4145) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Make `START REPLICATED SENDS` command start replicated sends. [#4229](https://github.com/yandex/ClickHouse/pull/4229) ([nvartolomei](https://github.com/nvartolomei))
|
||||||
|
* Fixed `Not found column` for duplicate columns in JOIN ON section. [#4279](https://github.com/yandex/ClickHouse/pull/4279) ([Artem Zuikov](https://github.com/4ertus2))
|
||||||
|
* Now `/etc/ssl` is used as default directory with SSL certificates. [#4167](https://github.com/yandex/ClickHouse/pull/4167) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Fixed crash on dictionary reload if dictionary not available. [#4188](https://github.com/yandex/ClickHouse/pull/4188) ([proller](https://github.com/proller))
|
||||||
|
* Fixed bug with incorrect `Date` and `DateTime` comparison. [#4237](https://github.com/yandex/ClickHouse/pull/4237) ([valexey](https://github.com/valexey))
|
||||||
|
* Fixed incorrect result when `Date` and `DateTime` arguments are used in branches of conditional operator (function `if`). Added generic case for function `if`. [#4243](https://github.com/yandex/ClickHouse/pull/4243) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
|
||||||
|
## ClickHouse release 19.1.6, 2019-01-24
|
||||||
|
|
||||||
|
### New Features
|
||||||
|
|
||||||
|
* Custom per column compression codecs for tables. [#3899](https://github.com/yandex/ClickHouse/pull/3899) [#4111](https://github.com/yandex/ClickHouse/pull/4111) ([alesapin](https://github.com/alesapin), [Winter Zhang](https://github.com/zhang2014), [Anatoly](https://github.com/Sindbag))
|
||||||
|
* Added compression codec `Delta`. [#4052](https://github.com/yandex/ClickHouse/pull/4052) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Allow to `ALTER` compression codecs. [#4054](https://github.com/yandex/ClickHouse/pull/4054) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Added functions `left`, `right`, `trim`, `ltrim`, `rtrim`, `timestampadd`, `timestampsub` for SQL standard compatibility. [#3826](https://github.com/yandex/ClickHouse/pull/3826) ([Ivan Blinkov](https://github.com/blinkov))
|
||||||
|
* Support for write in `HDFS` tables and `hdfs` table function. [#4084](https://github.com/yandex/ClickHouse/pull/4084) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Added functions to search for multiple constant strings from big haystack: `multiPosition`, `multiSearch` ,`firstMatch` also with `-UTF8`, `-CaseInsensitive`, and `-CaseInsensitiveUTF8` variants. [#4053](https://github.com/yandex/ClickHouse/pull/4053) ([Danila Kutenin](https://github.com/danlark1))
|
||||||
|
* Pruning of unused shards if `SELECT` query filters by sharding key (setting `optimize_skip_unused_shards`). [#3851](https://github.com/yandex/ClickHouse/pull/3851) ([Gleb Kanterov](https://github.com/kanterov), [Ivan](https://github.com/abyss7))
|
||||||
|
* Allow `Kafka` engine to ignore some number of parsing errors per block. [#4094](https://github.com/yandex/ClickHouse/pull/4094) ([Ivan](https://github.com/abyss7))
|
||||||
|
* Added support for `CatBoost` multiclass models evaluation. Function `modelEvaluate` returns tuple with per-class raw predictions for multiclass models. `libcatboostmodel.so` should be built with [#607](https://github.com/catboost/catboost/pull/607). [#3959](https://github.com/yandex/ClickHouse/pull/3959) ([KochetovNicolai](https://github.com/KochetovNicolai))
|
||||||
|
* Added functions `filesystemAvailable`, `filesystemFree`, `filesystemCapacity`. [#4097](https://github.com/yandex/ClickHouse/pull/4097) ([Boris Granveaud](https://github.com/bgranvea))
|
||||||
|
* Added hashing functions `xxHash64` and `xxHash32`. [#3905](https://github.com/yandex/ClickHouse/pull/3905) ([filimonov](https://github.com/filimonov))
|
||||||
|
* Added `gccMurmurHash` hashing function (GCC flavoured Murmur hash) which uses the same hash seed as [gcc](https://github.com/gcc-mirror/gcc/blob/41d6b10e96a1de98e90a7c0378437c3255814b16/libstdc%2B%2B-v3/include/bits/functional_hash.h#L191) [#4000](https://github.com/yandex/ClickHouse/pull/4000) ([sundyli](https://github.com/sundy-li))
|
||||||
|
* Added hashing functions `javaHash`, `hiveHash`. [#3811](https://github.com/yandex/ClickHouse/pull/3811) ([shangshujie365](https://github.com/shangshujie365))
|
||||||
|
* Added table function `remoteSecure`. Function works as `remote`, but uses secure connection. [#4088](https://github.com/yandex/ClickHouse/pull/4088) ([proller](https://github.com/proller))
|
||||||
|
|
||||||
|
|
||||||
|
### Experimental features
|
||||||
|
|
||||||
|
* Added multiple JOINs emulation (`allow_experimental_multiple_joins_emulation` setting). [#3946](https://github.com/yandex/ClickHouse/pull/3946) ([Artem Zuikov](https://github.com/4ertus2))
|
||||||
|
|
||||||
|
|
||||||
|
### Bug Fixes
|
||||||
|
|
||||||
|
* Make `compiled_expression_cache_size` setting limited by default to lower memory consumption. [#4041](https://github.com/yandex/ClickHouse/pull/4041) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Fix a bug that led to hangups in threads that perform ALTERs of Replicated tables and in the thread that updates configuration from ZooKeeper. [#2947](https://github.com/yandex/ClickHouse/issues/2947) [#3891](https://github.com/yandex/ClickHouse/issues/3891) [#3934](https://github.com/yandex/ClickHouse/pull/3934) ([Alex Zatelepin](https://github.com/ztlpn))
|
||||||
|
* Fixed a race condition when executing a distributed ALTER task. The race condition led to more than one replica trying to execute the task and all replicas except one failing with a ZooKeeper error. [#3904](https://github.com/yandex/ClickHouse/pull/3904) ([Alex Zatelepin](https://github.com/ztlpn))
|
||||||
|
* Fix a bug when `from_zk` config elements weren't refreshed after a request to ZooKeeper timed out. [#2947](https://github.com/yandex/ClickHouse/issues/2947) [#3947](https://github.com/yandex/ClickHouse/pull/3947) ([Alex Zatelepin](https://github.com/ztlpn))
|
||||||
|
* Fix bug with wrong prefix for IPv4 subnet masks. [#3945](https://github.com/yandex/ClickHouse/pull/3945) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Fixed crash (`std::terminate`) in rare cases when a new thread cannot be created due to exhausted resources. [#3956](https://github.com/yandex/ClickHouse/pull/3956) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Fix bug when in `remote` table function execution when wrong restrictions were used for in `getStructureOfRemoteTable`. [#4009](https://github.com/yandex/ClickHouse/pull/4009) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Fix a leak of netlink sockets. They were placed in a pool where they were never deleted and new sockets were created at the start of a new thread when all current sockets were in use. [#4017](https://github.com/yandex/ClickHouse/pull/4017) ([Alex Zatelepin](https://github.com/ztlpn))
|
||||||
|
* Fix bug with closing `/proc/self/fd` directory earlier than all fds were read from `/proc` after forking `odbc-bridge` subprocess. [#4120](https://github.com/yandex/ClickHouse/pull/4120) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Fixed String to UInt monotonic conversion in case of usage String in primary key. [#3870](https://github.com/yandex/ClickHouse/pull/3870) ([Winter Zhang](https://github.com/zhang2014))
|
||||||
|
* Fixed error in calculation of integer conversion function monotonicity. [#3921](https://github.com/yandex/ClickHouse/pull/3921) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Fixed segfault in `arrayEnumerateUniq`, `arrayEnumerateDense` functions in case of some invalid arguments. [#3909](https://github.com/yandex/ClickHouse/pull/3909) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Fix UB in StorageMerge. [#3910](https://github.com/yandex/ClickHouse/pull/3910) ([Amos Bird](https://github.com/amosbird))
|
||||||
|
* Fixed segfault in functions `addDays`, `subtractDays`. [#3913](https://github.com/yandex/ClickHouse/pull/3913) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Fixed error: functions `round`, `floor`, `trunc`, `ceil` may return bogus result when executed on integer argument and large negative scale. [#3914](https://github.com/yandex/ClickHouse/pull/3914) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Fixed a bug induced by 'kill query sync' which leads to a core dump. [#3916](https://github.com/yandex/ClickHouse/pull/3916) ([muVulDeePecker](https://github.com/fancyqlx))
|
||||||
|
* Fix bug with long delay after empty replication queue. [#3928](https://github.com/yandex/ClickHouse/pull/3928) [#3932](https://github.com/yandex/ClickHouse/pull/3932) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Fixed excessive memory usage in case of inserting into table with `LowCardinality` primary key. [#3955](https://github.com/yandex/ClickHouse/pull/3955) ([KochetovNicolai](https://github.com/KochetovNicolai))
|
||||||
|
* Fixed `LowCardinality` serialization for `Native` format in case of empty arrays. [#3907](https://github.com/yandex/ClickHouse/issues/3907) [#4011](https://github.com/yandex/ClickHouse/pull/4011) ([KochetovNicolai](https://github.com/KochetovNicolai))
|
||||||
|
* Fixed incorrect result while using distinct by single LowCardinality numeric column. [#3895](https://github.com/yandex/ClickHouse/issues/3895) [#4012](https://github.com/yandex/ClickHouse/pull/4012) ([KochetovNicolai](https://github.com/KochetovNicolai))
|
||||||
|
* Fixed specialized aggregation with LowCardinality key (in case when `compile` setting is enabled). [#3886](https://github.com/yandex/ClickHouse/pull/3886) ([KochetovNicolai](https://github.com/KochetovNicolai))
|
||||||
|
* Fix user and password forwarding for replicated tables queries. [#3957](https://github.com/yandex/ClickHouse/pull/3957) ([alesapin](https://github.com/alesapin)) ([小路](https://github.com/nicelulu))
|
||||||
|
* Fixed very rare race condition that can happen when listing tables in Dictionary database while reloading dictionaries. [#3970](https://github.com/yandex/ClickHouse/pull/3970) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Fixed incorrect result when HAVING was used with ROLLUP or CUBE. [#3756](https://github.com/yandex/ClickHouse/issues/3756) [#3837](https://github.com/yandex/ClickHouse/pull/3837) ([Sam Chou](https://github.com/reflection))
|
||||||
|
* Fixed column aliases for query with `JOIN ON` syntax and distributed tables. [#3980](https://github.com/yandex/ClickHouse/pull/3980) ([Winter Zhang](https://github.com/zhang2014))
|
||||||
|
* Fixed error in internal implementation of `quantileTDigest` (found by Artem Vakhrushev). This error never happens in ClickHouse and was relevant only for those who use ClickHouse codebase as a library directly. [#3935](https://github.com/yandex/ClickHouse/pull/3935) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
|
||||||
|
### Improvements
|
||||||
|
|
||||||
|
* Support for `IF NOT EXISTS` in `ALTER TABLE ADD COLUMN` statements along with `IF EXISTS` in `DROP/MODIFY/CLEAR/COMMENT COLUMN`. [#3900](https://github.com/yandex/ClickHouse/pull/3900) ([Boris Granveaud](https://github.com/bgranvea))
|
||||||
|
* Function `parseDateTimeBestEffort`: support for formats `DD.MM.YYYY`, `DD.MM.YY`, `DD-MM-YYYY`, `DD-Mon-YYYY`, `DD/Month/YYYY` and similar. [#3922](https://github.com/yandex/ClickHouse/pull/3922) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* `CapnProtoInputStream` now support jagged structures. [#4063](https://github.com/yandex/ClickHouse/pull/4063) ([Odin Hultgren Van Der Horst](https://github.com/Miniwoffer))
|
||||||
|
* Usability improvement: added a check that server process is started from the data directory's owner. Do not allow to start server from root if the data belongs to non-root user. [#3785](https://github.com/yandex/ClickHouse/pull/3785) ([sergey-v-galtsev](https://github.com/sergey-v-galtsev))
|
||||||
|
* Better logic of checking required columns during analysis of queries with JOINs. [#3930](https://github.com/yandex/ClickHouse/pull/3930) ([Artem Zuikov](https://github.com/4ertus2))
|
||||||
|
* Decreased the number of connections in case of large number of Distributed tables in a single server. [#3726](https://github.com/yandex/ClickHouse/pull/3726) ([Winter Zhang](https://github.com/zhang2014))
|
||||||
|
* Supported totals row for `WITH TOTALS` query for ODBC driver. [#3836](https://github.com/yandex/ClickHouse/pull/3836) ([Maksim Koritckiy](https://github.com/nightweb))
|
||||||
|
* Allowed to use `Enum`s as integers inside if function. [#3875](https://github.com/yandex/ClickHouse/pull/3875) ([Ivan](https://github.com/abyss7))
|
||||||
|
* Added `low_cardinality_allow_in_native_format` setting. If disabled, do not use `LowCadrinality` type in `Native` format. [#3879](https://github.com/yandex/ClickHouse/pull/3879) ([KochetovNicolai](https://github.com/KochetovNicolai))
|
||||||
|
* Removed some redundant objects from compiled expressions cache to lower memory usage. [#4042](https://github.com/yandex/ClickHouse/pull/4042) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Add check that `SET send_logs_level = 'value'` query accept appropriate value. [#3873](https://github.com/yandex/ClickHouse/pull/3873) ([Sabyanin Maxim](https://github.com/s-mx))
|
||||||
|
* Fixed data type check in type conversion functions. [#3896](https://github.com/yandex/ClickHouse/pull/3896) ([Winter Zhang](https://github.com/zhang2014))
|
||||||
|
|
||||||
|
### Performance Improvements
|
||||||
|
|
||||||
|
* Add a MergeTree setting `use_minimalistic_part_header_in_zookeeper`. If enabled, Replicated tables will store compact part metadata in a single part znode. This can dramatically reduce ZooKeeper snapshot size (especially if the tables have a lot of columns). Note that after enabling this setting you will not be able to downgrade to a version that doesn't support it. [#3960](https://github.com/yandex/ClickHouse/pull/3960) ([Alex Zatelepin](https://github.com/ztlpn))
|
||||||
|
* Add an DFA-based implementation for functions `sequenceMatch` and `sequenceCount` in case pattern doesn't contain time. [#4004](https://github.com/yandex/ClickHouse/pull/4004) ([Léo Ercolanelli](https://github.com/ercolanelli-leo))
|
||||||
|
* Performance improvement for integer numbers serialization. [#3968](https://github.com/yandex/ClickHouse/pull/3968) ([Amos Bird](https://github.com/amosbird))
|
||||||
|
* Zero left padding PODArray so that -1 element is always valid and zeroed. It's used for branchless calculation of offsets. [#3920](https://github.com/yandex/ClickHouse/pull/3920) ([Amos Bird](https://github.com/amosbird))
|
||||||
|
* Reverted `jemalloc` version which lead to performance degradation. [#4018](https://github.com/yandex/ClickHouse/pull/4018) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
|
||||||
|
### Backward Incompatible Changes
|
||||||
|
|
||||||
|
* Removed undocumented feature `ALTER MODIFY PRIMARY KEY` because it was superseded by the `ALTER MODIFY ORDER BY` command. [#3887](https://github.com/yandex/ClickHouse/pull/3887) ([Alex Zatelepin](https://github.com/ztlpn))
|
||||||
|
* Removed function `shardByHash`. [#3833](https://github.com/yandex/ClickHouse/pull/3833) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Forbid using scalar subqueries with result of type `AggregateFunction`. [#3865](https://github.com/yandex/ClickHouse/pull/3865) ([Ivan](https://github.com/abyss7))
|
||||||
|
|
||||||
|
### Build/Testing/Packaging Improvements
|
||||||
|
|
||||||
|
* Added support for PowerPC (`ppc64le`) build. [#4132](https://github.com/yandex/ClickHouse/pull/4132) ([Danila Kutenin](https://github.com/danlark1))
|
||||||
|
* Stateful functional tests are run on public available dataset. [#3969](https://github.com/yandex/ClickHouse/pull/3969) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Fixed error when the server cannot start with the `bash: /usr/bin/clickhouse-extract-from-config: Operation not permitted` message within Docker or systemd-nspawn. [#4136](https://github.com/yandex/ClickHouse/pull/4136) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Updated `rdkafka` library to v1.0.0-RC5. Used cppkafka instead of raw C interface. [#4025](https://github.com/yandex/ClickHouse/pull/4025) ([Ivan](https://github.com/abyss7))
|
||||||
|
* Updated `mariadb-client` library. Fixed one of issues found by UBSan. [#3924](https://github.com/yandex/ClickHouse/pull/3924) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Some fixes for UBSan builds. [#3926](https://github.com/yandex/ClickHouse/pull/3926) [#3021](https://github.com/yandex/ClickHouse/pull/3021) [#3948](https://github.com/yandex/ClickHouse/pull/3948) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Added per-commit runs of tests with UBSan build.
|
||||||
|
* Added per-commit runs of PVS-Studio static analyzer.
|
||||||
|
* Fixed bugs found by PVS-Studio. [#4013](https://github.com/yandex/ClickHouse/pull/4013) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Fixed glibc compatibility issues. [#4100](https://github.com/yandex/ClickHouse/pull/4100) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Move Docker images to 18.10 and add compatibility file for glibc >= 2.28 [#3965](https://github.com/yandex/ClickHouse/pull/3965) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Add env variable if user don't want to chown directories in server Docker image. [#3967](https://github.com/yandex/ClickHouse/pull/3967) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Enabled most of the warnings from `-Weverything` in clang. Enabled `-Wpedantic`. [#3986](https://github.com/yandex/ClickHouse/pull/3986) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Added a few more warnings that are available only in clang 8. [#3993](https://github.com/yandex/ClickHouse/pull/3993) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Link to `libLLVM` rather than to individual LLVM libs when using shared linking. [#3989](https://github.com/yandex/ClickHouse/pull/3989) ([Orivej Desh](https://github.com/orivej))
|
||||||
|
* Added sanitizer variables for test images. [#4072](https://github.com/yandex/ClickHouse/pull/4072) ([alesapin](https://github.com/alesapin))
|
||||||
|
* `clickhouse-server` debian package will recommend `libcap2-bin` package to use `setcap` tool for setting capabilities. This is optional. [#4093](https://github.com/yandex/ClickHouse/pull/4093) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Improved compilation time, fixed includes. [#3898](https://github.com/yandex/ClickHouse/pull/3898) ([proller](https://github.com/proller))
|
||||||
|
* Added performance tests for hash functions. [#3918](https://github.com/yandex/ClickHouse/pull/3918) ([filimonov](https://github.com/filimonov))
|
||||||
|
* Fixed cyclic library dependences. [#3958](https://github.com/yandex/ClickHouse/pull/3958) ([proller](https://github.com/proller))
|
||||||
|
* Improved compilation with low available memory. [#4030](https://github.com/yandex/ClickHouse/pull/4030) ([proller](https://github.com/proller))
|
||||||
|
* Added test script to reproduce performance degradation in `jemalloc`. [#4036](https://github.com/yandex/ClickHouse/pull/4036) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Fixed misspells in comments and string literals under `dbms`. [#4122](https://github.com/yandex/ClickHouse/pull/4122) ([maiha](https://github.com/maiha))
|
||||||
|
* Fixed typos in comments. [#4089](https://github.com/yandex/ClickHouse/pull/4089) ([Evgenii Pravda](https://github.com/kvinty))
|
||||||
|
|
||||||
|
|
||||||
## ClickHouse release 18.16.1, 2018-12-21
|
## ClickHouse release 18.16.1, 2018-12-21
|
||||||
|
|
||||||
### Bug fixes:
|
### Bug fixes:
|
||||||
|
|||||||
286
CHANGELOG_RU.md
286
CHANGELOG_RU.md
@ -1,3 +1,289 @@
|
|||||||
|
## ClickHouse release 19.3.5, 2019-02-21
|
||||||
|
|
||||||
|
### Исправления ошибок:
|
||||||
|
|
||||||
|
* Исправлена ошибка обработки длинных http-запросов на вставку на стороне сервера. [#4454](https://github.com/yandex/ClickHouse/pull/4454) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Исправлена обратная несовместимость со старыми версиями, появившаяся из-за некорректной реализации настройки `send_logs_level`. [#4445](https://github.com/yandex/ClickHouse/pull/4445) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Исправлена обратная несовместимость табличной функции `remote`, появившаяся из-за добавления комментариев колонок. [#4446](https://github.com/yandex/ClickHouse/pull/4446) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
|
||||||
|
## ClickHouse release 19.3.4, 2019-02-16
|
||||||
|
|
||||||
|
### Улучшения:
|
||||||
|
|
||||||
|
* При выполнении запроса `ATTACH TABLE` при проверке ограничений на используемую память теперь не учитывается память, занимаемая индексом таблицы. Это позволяет избежать ситуации, когда невозможно сделать `ATTACH TABLE` после соответствующего `DETACH TABLE`. [#4396](https://github.com/yandex/ClickHouse/pull/4396) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Немного увеличены ограничения на максимальный размер строки и массива, полученные от ZooKeeper. Это позволяет продолжать работу после увеличения настройки ZooKeeper `CLIENT_JVMFLAGS=-Djute.maxbuffer=...`. [#4398](https://github.com/yandex/ClickHouse/pull/4398) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Теперь реплику, отключенную на длительный период, можно восстановить, даже если в её очереди скопилось огромное число записей. [#4399](https://github.com/yandex/ClickHouse/pull/4399) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Для вторичных индексов типа `set` добавлен обязательный параметр (максимальное число хранимых значений). [#4386](https://github.com/yandex/ClickHouse/pull/4386) ([Nikita Vasilev](https://github.com/nikvas0))
|
||||||
|
|
||||||
|
### Исправления ошибок:
|
||||||
|
|
||||||
|
* Исправлен неверный результат запроса с модификатором `WITH ROLLUP` при группировке по единственному столбцу типа `LowCardinality`. [#4384](https://github.com/yandex/ClickHouse/pull/4384) ([Nikolai Kochetov](https://github.com/KochetovNicolai))
|
||||||
|
* Исправлена ошибка во вторичном индексе типа `set` (гранулы, в которых было больше, чем `max_rows` строк, игнорировались). [#4386](https://github.com/yandex/ClickHouse/pull/4386) ([Nikita Vasilev](https://github.com/nikvas0))
|
||||||
|
* Исправлена подстановка alias-ов в запросах с подзапросом, содержащим этот же alias ([#4110](https://github.com/yandex/ClickHouse/issues/4110)). [#4351](https://github.com/yandex/ClickHouse/pull/4351) ([Artem Zuikov](https://github.com/4ertus2))
|
||||||
|
|
||||||
|
### Улучшения сборки/тестирования/пакетирования:
|
||||||
|
|
||||||
|
* Множество исправлений для сборки под FreeBSD. [#4397](https://github.com/yandex/ClickHouse/pull/4397) ([proller](https://github.com/proller))
|
||||||
|
* Возможность запускать `clickhouse-server` для stateless тестов из docker-образа. [#4347](https://github.com/yandex/ClickHouse/pull/4347) ([Vasily Nemkov](https://github.com/Enmk))
|
||||||
|
|
||||||
|
## ClickHouse release 19.3.3, 2019-02-13
|
||||||
|
|
||||||
|
### Новые возможности:
|
||||||
|
|
||||||
|
* Добавлен запрос `KILL MUTATION`, который позволяет удалять мутации, которые по какой-то причине не могут выполниться. В таблицу `system.mutations` для облегчения диагностики добавлены столбцы `latest_failed_part`, `latest_fail_time`, `latest_fail_reason`. [#4287](https://github.com/yandex/ClickHouse/pull/4287) ([Alex Zatelepin](https://github.com/ztlpn))
|
||||||
|
* Добавлена агрегатная функция `entropy`, которая вычисляет энтропию Шеннона. [#4238](https://github.com/yandex/ClickHouse/pull/4238) ([Quid37](https://github.com/Quid37))
|
||||||
|
* Добавлена обобщённая реализация функции `arrayWithConstant`. [#4322](https://github.com/yandex/ClickHouse/pull/4322) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Добавлен оператор сравнения `NOT BETWEEN`. [#4228](https://github.com/yandex/ClickHouse/pull/4228) ([Dmitry Naumov](https://github.com/nezed))
|
||||||
|
* Добавлена функция `sumMapFiltered` - вариант `sumMap`, позволяющий указать набор ключей, по которым будет производиться суммирование. [#4129](https://github.com/yandex/ClickHouse/pull/4129) ([Léo Ercolanelli](https://github.com/ercolanelli-leo))
|
||||||
|
* Добавлена функция `sumMapWithOverflow`. [#4151](https://github.com/yandex/ClickHouse/pull/4151) ([Léo Ercolanelli](https://github.com/ercolanelli-leo))
|
||||||
|
* Добавлена поддержка `Nullable` типов в табличной функции `mysql`. [#4198](https://github.com/yandex/ClickHouse/pull/4198) ([Emmanuel Donin de Rosière](https://github.com/edonin))
|
||||||
|
* Добавлена поддержка произвольных константных выражений в секции `LIMIT`. [#4246](https://github.com/yandex/ClickHouse/pull/4246) ([k3box](https://github.com/k3box))
|
||||||
|
* Добавлена агрегатная функция `topKWeighted` - вариант `topK`, позволяющий задавать (целый неотрицательный) вес добавляемого значения. [#4245](https://github.com/yandex/ClickHouse/pull/4245) ([Andrew Golman](https://github.com/andrewgolman))
|
||||||
|
* Движок `Join` теперь поддерживает настройку `join_overwrite`, которая позволяет перезаписывать значения для существующих ключей. [#3973](https://github.com/yandex/ClickHouse/pull/3973) ([Amos Bird](https://github.com/amosbird))
|
||||||
|
* Добавлена функция `toStartOfInterval`. [#4304](https://github.com/yandex/ClickHouse/pull/4304) ([Vitaly Baranov](https://github.com/vitlibar))
|
||||||
|
* Добавлена функция `toStartOfTenMinutes`. [#4298](https://github.com/yandex/ClickHouse/pull/4298) ([Vitaly Baranov](https://github.com/vitlibar))
|
||||||
|
* Добавлен формат `RowBinaryWithNamesAndTypes`. [#4200](https://github.com/yandex/ClickHouse/pull/4200) ([Oleg V. Kozlyuk](https://github.com/DarkWanderer))
|
||||||
|
* Добавлены типы `IPv4` и `IPv6`. Более эффективная реализация функций `IPv*`. [#3669](https://github.com/yandex/ClickHouse/pull/3669) ([Vasily Nemkov](https://github.com/Enmk))
|
||||||
|
* Добавлен выходной формат `Protobuf`. [#4005](https://github.com/yandex/ClickHouse/pull/4005) [#4158](https://github.com/yandex/ClickHouse/pull/4158) ([Vitaly Baranov](https://github.com/vitlibar))
|
||||||
|
* В HTTP-интерфейсе добавлена поддержка алгоритма сжатия brotli для вставляемых данных. [#4235](https://github.com/yandex/ClickHouse/pull/4235) ([Mikhail](https://github.com/fandyushin))
|
||||||
|
* Клиент командной строки теперь подсказывает правильное имя, если пользователь опечатался в названии функции. [#4239](https://github.com/yandex/ClickHouse/pull/4239) ([Danila Kutenin](https://github.com/danlark1))
|
||||||
|
* В HTTP-ответ сервера добавлен заголовок `Query-Id`. [#4231](https://github.com/yandex/ClickHouse/pull/4231) ([Mikhail](https://github.com/fandyushin))
|
||||||
|
|
||||||
|
### Экспериментальные возможности:
|
||||||
|
|
||||||
|
* Добавлена поддержка вторичных индексов типа `minmax` и `set` для таблиц семейства MergeTree (позволяют быстро пропускать целые блоки данных). [#4143](https://github.com/yandex/ClickHouse/pull/4143) ([Nikita Vasilev](https://github.com/nikvas0))
|
||||||
|
* Добавлена поддержка преобразования `CROSS JOIN` в `INNER JOIN`, если это возможно. [#4221](https://github.com/yandex/ClickHouse/pull/4221) [#4266](https://github.com/yandex/ClickHouse/pull/4266) ([Artem Zuikov](https://github.com/4ertus2))
|
||||||
|
|
||||||
|
### Исправления ошибок:
|
||||||
|
|
||||||
|
* Исправлена ошибка `Not found column` для случая дублирующихся столбцов в секции `JOIN ON`. [#4279](https://github.com/yandex/ClickHouse/pull/4279) ([Artem Zuikov](https://github.com/4ertus2))
|
||||||
|
* Команда `START REPLICATED SENDS` теперь действительно включает посылку кусков данных при репликации. [#4229](https://github.com/yandex/ClickHouse/pull/4229) ([nvartolomei](https://github.com/nvartolomei))
|
||||||
|
* Исправлена агрегация столбцов типа `Array(LowCardinality)`. [#4055](https://github.com/yandex/ClickHouse/pull/4055) ([KochetovNicolai](https://github.com/KochetovNicolai))
|
||||||
|
* Исправлена ошибка, приводившая к тому, что при исполнении запроса `INSERT ... SELECT ... FROM file(...)` терялась первая строчка файла, если он был в формате `CSVWithNames` или `TSVWIthNames`. [#4297](https://github.com/yandex/ClickHouse/pull/4297) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Исправлено падение при перезагрузке внешнего словаря, если словарь недоступен. Ошибка возникла в 19.1.6. [#4188](https://github.com/yandex/ClickHouse/pull/4188) ([proller](https://github.com/proller))
|
||||||
|
* Исправлен неверный результат `ALL JOIN`, если в правой таблице присутствуют дубликаты ключа join. [#4184](https://github.com/yandex/ClickHouse/pull/4184) ([Artem Zuikov](https://github.com/4ertus2))
|
||||||
|
* Исправлено падение сервера при включённой опции `use_uncompressed_cache`, а также исключение о неправильном размере разжатых данных. [#4186](https://github.com/yandex/ClickHouse/pull/4186) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Исправлена ошибка, приводящая к неправильному результату сравнения больших (не помещающихся в Int16) дат при включённой настройке `compile_expressions`. [#4341](https://github.com/yandex/ClickHouse/pull/4341) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Исправлен бесконечный цикл при запросе из табличной функции `numbers(0)`. [#4280](https://github.com/yandex/ClickHouse/pull/4280) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Временно отключён pushdown предикатов в подзапрос, если он содержит `ORDER BY`. [#3890](https://github.com/yandex/ClickHouse/pull/3890) ([Winter Zhang](https://github.com/zhang2014))
|
||||||
|
* Исправлена ошибка `Illegal instruction` при использовании функций для работы с base64 на старых CPU. Ошибка проявлялась только, если ClickHouse был скомпилирован с gcc-8. [#4275](https://github.com/yandex/ClickHouse/pull/4275) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Исправлена ошибка `No message received` при запросах к PostgreSQL через ODBC-драйвер и TLS-соединение, исправлен segfault при использовании MySQL через ODBC-драйвер. [#4170](https://github.com/yandex/ClickHouse/pull/4170) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Исправлен неверный результат при использовании значений типа `Date` или `DateTime` в ветвях условного оператора (функции `if`). Функция `if` теперь работает для произвольного типа значений в ветвях. [#4243](https://github.com/yandex/ClickHouse/pull/4243) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Словари с источником из локального ClickHouse теперь исполняются локально, а не используя TCP-соединение. [#4166](https://github.com/yandex/ClickHouse/pull/4166) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Исправлено зависание запросов к таблице с движком `File` после того, как `SELECT` из этой таблицы завершился с ошибкой `No such file or directory`. [#4161](https://github.com/yandex/ClickHouse/pull/4161) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Исправлена ошибка, из-за которой при запросе к таблице `system.tables` могло возникать исключение `table doesn't exist`. [#4313](https://github.com/yandex/ClickHouse/pull/4313) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Исправлена ошибка, приводившая к падению `clickhouse-client` в интерактивном режиме, если успеть выйти из него во время загрузки подсказок командной строки. [#4317](https://github.com/yandex/ClickHouse/pull/4317) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Исправлена ошибка, приводившая к неверным результатам исполнения мутаций, содержащих оператор `IN`. [#4099](https://github.com/yandex/ClickHouse/pull/4099) ([Alex Zatelepin](https://github.com/ztlpn))
|
||||||
|
* Исправлена ошибка, из-за которой, если была создана база данных с движком `Dictionary`, все словари загружались при старте сервера, а словари с источником из локального ClickHouse не могли загрузиться. [#4255](https://github.com/yandex/ClickHouse/pull/4255) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Исправлено повторное создание таблиц с системными логами (`system.query_log`, `system.part_log`) при остановке сервера. [#4254](https://github.com/yandex/ClickHouse/pull/4254) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Исправлен вывод типа возвращаемого значения, а также использование блокировок в функции `joinGet`. [#4153](https://github.com/yandex/ClickHouse/pull/4153) ([Amos Bird](https://github.com/amosbird))
|
||||||
|
* Исправлено падение сервера при использовании настройки `allow_experimental_multiple_joins_emulation`. [52de2c](https://github.com/yandex/ClickHouse/commit/52de2cd927f7b5257dd67e175f0a5560a48840d0) ([Artem Zuikov](https://github.com/4ertus2))
|
||||||
|
* Исправлено некорректное сравнение значений типа `Date` и `DateTime`. [#4237](https://github.com/yandex/ClickHouse/pull/4237) ([valexey](https://github.com/valexey))
|
||||||
|
* Исправлена ошибка, проявлявшаяся при fuzz-тестировании с undefined behaviour-санитайзером: добавлена проверка типов параметров для семейства функций `quantile*Weighted`. [#4145](https://github.com/yandex/ClickHouse/pull/4145) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Исправлена редкая ошибка, из-за которой при удалении старых кусков данных может возникать ошибка `File not found`. [#4378](https://github.com/yandex/ClickHouse/pull/4378) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Исправлена установка пакета при отсутствующем файле /etc/clickhouse-server/config.xml. [#4343](https://github.com/yandex/ClickHouse/pull/4343) ([proller](https://github.com/proller))
|
||||||
|
|
||||||
|
### Улучшения сборки/тестирования/пакетирования:
|
||||||
|
|
||||||
|
* При установке Debian-пакета символическая ссылка /etc/clickhouse-server/preprocessed теперь создаётся, учитывая пути, прописанные в конфигурационном файле. [#4205](https://github.com/yandex/ClickHouse/pull/4205) ([proller](https://github.com/proller))
|
||||||
|
* Исправления сборки под FreeBSD. [#4225](https://github.com/yandex/ClickHouse/pull/4225) ([proller](https://github.com/proller))
|
||||||
|
* Добавлена возможность создавать, заполнять и удалять таблицы в тестах производительности. [#4220](https://github.com/yandex/ClickHouse/pull/4220) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Добавлен скрипт для поиска дублирующихся include-директив в исходных файлах. [#4326](https://github.com/yandex/ClickHouse/pull/4326) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* В тестах производительности добавлена возможность запускать запросы по номеру. [#4264](https://github.com/yandex/ClickHouse/pull/4264) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Пакет с debug-символами добавлен в список рекомендованных для основного пакета. [#4274](https://github.com/yandex/ClickHouse/pull/4274) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Рефакторинг утилиты performance-test. Улучшено логирование и обработка сигналов. [#4171](https://github.com/yandex/ClickHouse/pull/4171) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Задокументирован анонимизированный датасет Яндекс.Метрики. [#4164](https://github.com/yandex/ClickHouse/pull/4164) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Добавлен инструмент для конвертирования кусков данных таблиц, созданных с использованием старого синтаксиса с помесячным партиционированием, в новый формат. [#4195](https://github.com/yandex/ClickHouse/pull/4195) ([Alex Zatelepin](https://github.com/ztlpn))
|
||||||
|
* Добавлена документация для двух датасетов, загруженных в s3. [#4144](https://github.com/yandex/ClickHouse/pull/4144) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Добавлен инструмент, собирающий changelog из описаний pull request-ов. [#4169](https://github.com/yandex/ClickHouse/pull/4169) [#4173](https://github.com/yandex/ClickHouse/pull/4173) ([KochetovNicolai](https://github.com/KochetovNicolai)) ([KochetovNicolai](https://github.com/KochetovNicolai))
|
||||||
|
* Добавлен puppet-модуль для Clickhouse. [#4182](https://github.com/yandex/ClickHouse/pull/4182) ([Maxim Fedotov](https://github.com/MaxFedotov))
|
||||||
|
* Добавлена документация для нескольких недокументированных функций. [#4168](https://github.com/yandex/ClickHouse/pull/4168) ([Winter Zhang](https://github.com/zhang2014))
|
||||||
|
* Исправления сборки под ARM. [#4210](https://github.com/yandex/ClickHouse/pull/4210)[#4306](https://github.com/yandex/ClickHouse/pull/4306) [#4291](https://github.com/yandex/ClickHouse/pull/4291) ([proller](https://github.com/proller)) ([proller](https://github.com/proller))
|
||||||
|
* Добавлена возможность запускать тесты словарей из `ctest`. [#4189](https://github.com/yandex/ClickHouse/pull/4189) ([proller](https://github.com/proller))
|
||||||
|
* Теперь директорией с SSL-сертификатами по умолчанию является `/etc/ssl`. [#4167](https://github.com/yandex/ClickHouse/pull/4167) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Добавлена проверка доступности SSE и AVX-инструкций на старте. [#4234](https://github.com/yandex/ClickHouse/pull/4234) ([Igr](https://github.com/igron99))
|
||||||
|
* Init-скрипт теперь дожидается, пока сервер запустится. [#4281](https://github.com/yandex/ClickHouse/pull/4281) ([proller](https://github.com/proller))
|
||||||
|
|
||||||
|
### Обратно несовместимые изменения:
|
||||||
|
|
||||||
|
* Удалена настройка `allow_experimental_low_cardinality_type`. Семейство типов данных `LowCardinality` готово для использования в production. [#4323](https://github.com/yandex/ClickHouse/pull/4323) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Размер кэша засечек и кэша разжатых блоков теперь уменьшается в зависимости от доступного объёма памяти. [#4240](https://github.com/yandex/ClickHouse/pull/4240) ([Lopatin Konstantin](https://github.com/k-lopatin))
|
||||||
|
* Для запроса `CREATE TABLE` добавлено ключевое слово `INDEX`. Имя столбца `index` теперь надо оборачивать в двойные или обратные кавычки: `` `index` ``. [#4143](https://github.com/yandex/ClickHouse/pull/4143) ([Nikita Vasilev](https://github.com/nikvas0))
|
||||||
|
* Функция `sumMap` теперь возвращает тип с большей областью значений вместо переполнения. Если необходимо старое поведение, следует использовать добавленную функцию `sumMapWithOverflow`. [#4151](https://github.com/yandex/ClickHouse/pull/4151) ([Léo Ercolanelli](https://github.com/ercolanelli-leo))
|
||||||
|
|
||||||
|
### Улучшения производительности:
|
||||||
|
|
||||||
|
* Для запросов без секции `LIMIT` вместо `std::sort` теперь используется `pdqsort`. [#4236](https://github.com/yandex/ClickHouse/pull/4236) ([Evgenii Pravda](https://github.com/kvinty))
|
||||||
|
* Теперь сервер переиспользует потоки для выполнения запросов из глобального пула потоков. В краевых случаях это влияет на производительность. [#4150](https://github.com/yandex/ClickHouse/pull/4150) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
|
||||||
|
### Улучшения:
|
||||||
|
|
||||||
|
* Теперь, если в нативном протоколе послать запрос `INSERT INTO tbl VALUES (....` (с данными в запросе), отдельно посылать разобранные данные для вставки не нужно. [#4301](https://github.com/yandex/ClickHouse/pull/4301) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Добавлена поддержка AIO для FreeBSD. [#4305](https://github.com/yandex/ClickHouse/pull/4305) ([urgordeadbeef](https://github.com/urgordeadbeef))
|
||||||
|
* Запрос `SELECT * FROM a JOIN b USING a, b` теперь возвращает столбцы `a` и `b` только из левой таблицы. [#4141](https://github.com/yandex/ClickHouse/pull/4141) ([Artem Zuikov](https://github.com/4ertus2))
|
||||||
|
* Добавлена опция командной строки `-C` для клиента, которая работает так же, как и опция `-c`. [#4232](https://github.com/yandex/ClickHouse/pull/4232) ([syominsergey](https://github.com/syominsergey))
|
||||||
|
* Если для опции `--password` клиента командной строки не указано значение, пароль запрашивается из стандартного входа. [#4230](https://github.com/yandex/ClickHouse/pull/4230) ([BSD_Conqueror](https://github.com/bsd-conqueror))
|
||||||
|
* Добавлена подсветка метасимволов в строковых литералах, содержащих выражения для оператора `LIKE` и регулярные выражения. [#4327](https://github.com/yandex/ClickHouse/pull/4327) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Добавлена отмена HTTP-запроса, если сокет клиента отваливается. [#4213](https://github.com/yandex/ClickHouse/pull/4213) ([nvartolomei](https://github.com/nvartolomei))
|
||||||
|
* Теперь сервер время от времени посылает пакеты Progress для поддержания соединения. [#4215](https://github.com/yandex/ClickHouse/pull/4215) ([Ivan](https://github.com/abyss7))
|
||||||
|
* Немного улучшено сообщение о причине, почему запрос OPTIMIZE не может быть исполнен (если включена настройка `optimize_throw_if_noop`). [#4294](https://github.com/yandex/ClickHouse/pull/4294) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Добавлена поддержка опции `--version` для `clickhouse-server`. [#4251](https://github.com/yandex/ClickHouse/pull/4251) ([Lopatin Konstantin](https://github.com/k-lopatin))
|
||||||
|
* Добавлена поддержка опции `--help/-h` для `clickhouse-server`. [#4233](https://github.com/yandex/ClickHouse/pull/4233) ([Yuriy Baranov](https://github.com/yurriy))
|
||||||
|
* Добавлена поддержка скалярных подзапросов, возвращающих состояние агрегатной функции. [#4348](https://github.com/yandex/ClickHouse/pull/4348) ([Nikolai Kochetov](https://github.com/KochetovNicolai))
|
||||||
|
* Уменьшено время ожидания завершения сервера и завершения запросов `ALTER`. [#4372](https://github.com/yandex/ClickHouse/pull/4372) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Добавлена информация о значении настройки `replicated_can_become_leader` в таблицу `system.replicas`. Добавлено логирование в случае, если реплика не собирается стать лидером. [#4379](https://github.com/yandex/ClickHouse/pull/4379) ([Alex Zatelepin](https://github.com/ztlpn))
|
||||||
|
|
||||||
|
## ClickHouse release 19.1.9, 2019-02-21
|
||||||
|
|
||||||
|
### Исправления ошибок:
|
||||||
|
|
||||||
|
* Исправлена обратная несовместимость со старыми версиями, появившаяся из-за некорректной реализации настройки `send_logs_level`. [#4445](https://github.com/yandex/ClickHouse/pull/4445) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Исправлена обратная несовместимость табличной функции `remote`, появившаяся из-за добавления комментариев колонок. [#4446](https://github.com/yandex/ClickHouse/pull/4446) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
|
||||||
|
## ClickHouse release 19.1.8, 2019-02-16
|
||||||
|
|
||||||
|
### Исправления ошибок:
|
||||||
|
|
||||||
|
* Исправлена установка пакета при отсутствующем файле /etc/clickhouse-server/config.xml. [#4343](https://github.com/yandex/ClickHouse/pull/4343) ([proller](https://github.com/proller))
|
||||||
|
|
||||||
|
## ClickHouse release 19.1.7, 2019-02-15
|
||||||
|
|
||||||
|
### Исправления ошибок:
|
||||||
|
|
||||||
|
* Исправлен вывод типа возвращаемого значения, а также использование блокировок в функции `joinGet`. [#4153](https://github.com/yandex/ClickHouse/pull/4153) ([Amos Bird](https://github.com/amosbird))
|
||||||
|
* Исправлено повторное создание таблиц с системными логами (`system.query_log`, `system.part_log`) при остановке сервера. [#4254](https://github.com/yandex/ClickHouse/pull/4254) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Исправлена ошибка, из-за которой, если была создана база данных с движком `Dictionary`, все словари загружались при старте сервера, а словари с источником из локального ClickHouse не могли загрузиться. [#4255](https://github.com/yandex/ClickHouse/pull/4255) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Исправлена ошибка, приводившая к неверным результатам исполнения мутаций, содержащих оператор `IN`. [#4099](https://github.com/yandex/ClickHouse/pull/4099) ([Alex Zatelepin](https://github.com/ztlpn))
|
||||||
|
* Исправлена ошибка, приводившая к падению `clickhouse-client` в интерактивном режиме, если успеть выйти из него во время загрузки подсказок командной строки. [#4317](https://github.com/yandex/ClickHouse/pull/4317) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Исправлена ошибка, из-за которой при запросе к таблице `system.tables` могло возникать исключение `table doesn't exist`. [#4313](https://github.com/yandex/ClickHouse/pull/4313) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Исправлено зависание запросов к таблице с движком `File` после того, как `SELECT` из этой таблицы завершился с ошибкой `No such file or directory`. [#4161](https://github.com/yandex/ClickHouse/pull/4161) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Словари с источником из локального ClickHouse теперь исполняются локально, а не используя TCP-соединение. [#4166](https://github.com/yandex/ClickHouse/pull/4166) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Исправлена ошибка `No message received` при запросах к PostgreSQL через ODBC-драйвер и TLS-соединение, исправлен segfault при использовании MySQL через ODBC-драйвер. [#4170](https://github.com/yandex/ClickHouse/pull/4170) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Временно отключён pushdown предикатов в подзапрос, если он содержит `ORDER BY`. [#3890](https://github.com/yandex/ClickHouse/pull/3890) ([Winter Zhang](https://github.com/zhang2014))
|
||||||
|
* Исправлен бесконечный цикл при запросе из табличной функции `numbers(0)`. [#4280](https://github.com/yandex/ClickHouse/pull/4280) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Исправлена ошибка, приводящая к неправильному результату сравнения больших (не помещающихся в Int16) дат при включённой настройке `compile_expressions`. [#4341](https://github.com/yandex/ClickHouse/pull/4341) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Исправлено падение сервера при включённой опции `uncompressed_cache`, а также исключение о неправильном размере разжатых данных. [#4186](https://github.com/yandex/ClickHouse/pull/4186) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Исправлен неверный результат `ALL JOIN`, если в правой таблице присутствуют дубликаты ключа join. [#4184](https://github.com/yandex/ClickHouse/pull/4184) ([Artem Zuikov](https://github.com/4ertus2))
|
||||||
|
* Исправлена ошибка, приводившая к тому, что при исполнении запроса `INSERT ... SELECT ... FROM file(...)` терялась первая строчка файла, если он был в формате `CSVWithNames` или `TSVWIthNames`. [#4297](https://github.com/yandex/ClickHouse/pull/4297) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Исправлена агрегация столбцов типа `Array(LowCardinality)`. [#4055](https://github.com/yandex/ClickHouse/pull/4055) ([KochetovNicolai](https://github.com/KochetovNicolai))
|
||||||
|
* При установке Debian-пакета символическая ссылка /etc/clickhouse-server/preprocessed теперь создаётся, учитывая пути, прописанные в конфигурационном файле. [#4205](https://github.com/yandex/ClickHouse/pull/4205) ([proller](https://github.com/proller))
|
||||||
|
* Исправлена ошибка, проявлявшаяся при fuzz-тестировании с undefined behaviour-санитайзером: добавлена проверка типов параметров для семейства функций `quantile*Weighted`. [#4145](https://github.com/yandex/ClickHouse/pull/4145) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Команда `START REPLICATED SENDS` теперь действительно включает посылку кусков данных при репликации. [#4229](https://github.com/yandex/ClickHouse/pull/4229) ([nvartolomei](https://github.com/nvartolomei))
|
||||||
|
* Исправлена ошибка `Not found column` для случая дублирующихся столбцов в секции `JOIN ON`. [#4279](https://github.com/yandex/ClickHouse/pull/4279) ([Artem Zuikov](https://github.com/4ertus2))
|
||||||
|
* Теперь директорией с SSL-сертификатами по умолчанию является `/etc/ssl`. [#4167](https://github.com/yandex/ClickHouse/pull/4167) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Исправлено падение при перезагрузке внешнего словаря, если словарь недоступен. Ошибка возникла в 19.1.6. [#4188](https://github.com/yandex/ClickHouse/pull/4188) ([proller](https://github.com/proller))
|
||||||
|
* Исправлено некорректное сравнение значений типа `Date` и `DateTime`. [#4237](https://github.com/yandex/ClickHouse/pull/4237) ([valexey](https://github.com/valexey))
|
||||||
|
* Исправлен неверный результат при использовании значений типа `Date` или `DateTime` в ветвях условного оператора (функции `if`). Функция `if` теперь работает для произвольного типа значений в ветвях. [#4243](https://github.com/yandex/ClickHouse/pull/4243) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
|
||||||
|
## ClickHouse release 19.1.6, 2019-01-24
|
||||||
|
|
||||||
|
### Новые возможности:
|
||||||
|
|
||||||
|
* Задание формата сжатия для отдельных столбцов. [#3899](https://github.com/yandex/ClickHouse/pull/3899) [#4111](https://github.com/yandex/ClickHouse/pull/4111) ([alesapin](https://github.com/alesapin), [Winter Zhang](https://github.com/zhang2014), [Anatoly](https://github.com/Sindbag))
|
||||||
|
* Формат сжатия `Delta`. [#4052](https://github.com/yandex/ClickHouse/pull/4052) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Изменение формата сжатия запросом `ALTER`. [#4054](https://github.com/yandex/ClickHouse/pull/4054) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Добавлены функции `left`, `right`, `trim`, `ltrim`, `rtrim`, `timestampadd`, `timestampsub` для совместимости со стандартом SQL. [#3826](https://github.com/yandex/ClickHouse/pull/3826) ([Ivan Blinkov](https://github.com/blinkov))
|
||||||
|
* Поддержка записи в движок `HDFS` и табличную функцию `hdfs`. [#4084](https://github.com/yandex/ClickHouse/pull/4084) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Добавлены функции поиска набора константных строк в тексте: `multiPosition`, `multiSearch` ,`firstMatch` также с суффиксами `-UTF8`, `-CaseInsensitive`, и `-CaseInsensitiveUTF8`. [#4053](https://github.com/yandex/ClickHouse/pull/4053) ([Danila Kutenin](https://github.com/danlark1))
|
||||||
|
* Пропуск неиспользуемых шардов в случае, если запрос `SELECT` содержит фильтрацию по ключу шардирования (настройка `optimize_skip_unused_shards`). [#3851](https://github.com/yandex/ClickHouse/pull/3851) ([Gleb Kanterov](https://github.com/kanterov), [Ivan](https://github.com/abyss7))
|
||||||
|
* Пропуск строк в случае ошибки парсинга для движка `Kafka` (настройка `kafka_skip_broken_messages`). [#4094](https://github.com/yandex/ClickHouse/pull/4094) ([Ivan](https://github.com/abyss7))
|
||||||
|
* Поддержка применения мультиклассовых моделей `CatBoost`. Функция `modelEvaluate` возвращает кортеж в случае использования мультиклассовой модели. `libcatboostmodel.so` should be built with [#607](https://github.com/catboost/catboost/pull/607). [#3959](https://github.com/yandex/ClickHouse/pull/3959) ([KochetovNicolai](https://github.com/KochetovNicolai))
|
||||||
|
* Добавлены функции `filesystemAvailable`, `filesystemFree`, `filesystemCapacity`. [#4097](https://github.com/yandex/ClickHouse/pull/4097) ([Boris Granveaud](https://github.com/bgranvea))
|
||||||
|
* Добавлены функции хеширования `xxHash64` и `xxHash32`. [#3905](https://github.com/yandex/ClickHouse/pull/3905) ([filimonov](https://github.com/filimonov))
|
||||||
|
* Добавлена функция хеширования `gccMurmurHash` (GCC flavoured Murmur hash), использующая те же hash seed, что и [gcc](https://github.com/gcc-mirror/gcc/blob/41d6b10e96a1de98e90a7c0378437c3255814b16/libstdc%2B%2B-v3/include/bits/functional_hash.h#L191) [#4000](https://github.com/yandex/ClickHouse/pull/4000) ([sundyli](https://github.com/sundy-li))
|
||||||
|
* Добавлены функции хеширования `javaHash`, `hiveHash`. [#3811](https://github.com/yandex/ClickHouse/pull/3811) ([shangshujie365](https://github.com/shangshujie365))
|
||||||
|
* Добавлена функция `remoteSecure`. Функция работает аналогично `remote`, но использует безопасное соединение. [#4088](https://github.com/yandex/ClickHouse/pull/4088) ([proller](https://github.com/proller))
|
||||||
|
|
||||||
|
|
||||||
|
### Экспериментальные возможности:
|
||||||
|
|
||||||
|
* Эмуляция запросов с несколькими секциями `JOIN` (настройка `allow_experimental_multiple_joins_emulation`). [#3946](https://github.com/yandex/ClickHouse/pull/3946) ([Artem Zuikov](https://github.com/4ertus2))
|
||||||
|
|
||||||
|
### Исправления ошибок:
|
||||||
|
|
||||||
|
* Ограничен размер кеша скомпилированных выражений в случае, если не указана настройка `compiled_expression_cache_size` для экономии потребляемой памяти. [#4041](https://github.com/yandex/ClickHouse/pull/4041) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Исправлена проблема зависания потоков, выполняющих запрос `ALTER` для таблиц семейства `Replicated`, а также потоков, обновляющих конфигурацию из ZooKeeper. [#2947](https://github.com/yandex/ClickHouse/issues/2947) [#3891](https://github.com/yandex/ClickHouse/issues/3891) [#3934](https://github.com/yandex/ClickHouse/pull/3934) ([Alex Zatelepin](https://github.com/ztlpn))
|
||||||
|
* Исправлен race condition в случае выполнения распределенной задачи запроса `ALTER`. Race condition приводил к состоянию, когда более чем одна реплика пыталась выполнить задачу, в результате чего все такие реплики, кроме одной, падали с ошибкой обращения к ZooKeeper. [#3904](https://github.com/yandex/ClickHouse/pull/3904) ([Alex Zatelepin](https://github.com/ztlpn))
|
||||||
|
* Исправлена проблема обновления настройки `from_zk`. Настройка, указанная в файле конфигурации, не обновлялась в случае, если запрос к ZooKeeper падал по timeout. [#2947](https://github.com/yandex/ClickHouse/issues/2947) [#3947](https://github.com/yandex/ClickHouse/pull/3947) ([Alex Zatelepin](https://github.com/ztlpn))
|
||||||
|
* Исправлена ошибка в вычислении сетевого префикса при указании IPv4 маски подсети. [#3945](https://github.com/yandex/ClickHouse/pull/3945) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Исправлено падение (`std::terminate`) в редком сценарии, когда новый поток не мог быть создан из-за нехватки ресурсов. [#3956](https://github.com/yandex/ClickHouse/pull/3956) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Исправлено падение табличной функции `remote` в случае, когда не удавалось получить структуру таблицы из-за ограничений пользователя. [#4009](https://github.com/yandex/ClickHouse/pull/4009) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Исправлена утечка сетевых сокетов. Сокеты создавались в пуле и никогда не закрывались. При создании потока, создавались новые сокеты в случае, если все доступные использовались. [#4017](https://github.com/yandex/ClickHouse/pull/4017) ([Alex Zatelepin](https://github.com/ztlpn))
|
||||||
|
* Исправлена проблема закрывания `/proc/self/fd` раньше, чем все файловые дескрипторы были прочитаны из `/proc` после создания процесса `odbc-bridge`. [#4120](https://github.com/yandex/ClickHouse/pull/4120) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Исправлен баг в монотонном преобразовании String в UInt в случае использования String в первичном ключе. [#3870](https://github.com/yandex/ClickHouse/pull/3870) ([Winter Zhang](https://github.com/zhang2014))
|
||||||
|
* Исправлен баг в вычислении монотонности функции преобразования типа целых значений. [#3921](https://github.com/yandex/ClickHouse/pull/3921) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Исправлено падение в функциях `arrayEnumerateUniq`, `arrayEnumerateDense` при передаче невалидных аргументов. [#3909](https://github.com/yandex/ClickHouse/pull/3909) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Исправлен undefined behavior в StorageMerge. [#3910](https://github.com/yandex/ClickHouse/pull/3910) ([Amos Bird](https://github.com/amosbird))
|
||||||
|
* Исправлено падение в функциях `addDays`, `subtractDays`. [#3913](https://github.com/yandex/ClickHouse/pull/3913) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Исправлена проблема, в результате которой функции `round`, `floor`, `trunc`, `ceil` могли возвращать неверный результат для отрицательных целочисленных аргументов с большим значением. [#3914](https://github.com/yandex/ClickHouse/pull/3914) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Исправлена проблема, в результате которой 'kill query sync' приводил к падению сервера. [#3916](https://github.com/yandex/ClickHouse/pull/3916) ([muVulDeePecker](https://github.com/fancyqlx))
|
||||||
|
* Исправлен баг, приводящий к большой задержке в случае пустой очереди репликации. [#3928](https://github.com/yandex/ClickHouse/pull/3928) [#3932](https://github.com/yandex/ClickHouse/pull/3932) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Исправлено избыточное использование памяти в случае вставки в таблицу с `LowCardinality` в первичном ключе. [#3955](https://github.com/yandex/ClickHouse/pull/3955) ([KochetovNicolai](https://github.com/KochetovNicolai))
|
||||||
|
* Исправлена сериализация пустых массивов типа `LowCardinality` для формата `Native`. [#3907](https://github.com/yandex/ClickHouse/issues/3907) [#4011](https://github.com/yandex/ClickHouse/pull/4011) ([KochetovNicolai](https://github.com/KochetovNicolai))
|
||||||
|
* Исправлен неверный результат в случае использования distinct для числового столбца `LowCardinality`. [#3895](https://github.com/yandex/ClickHouse/issues/3895) [#4012](https://github.com/yandex/ClickHouse/pull/4012) ([KochetovNicolai](https://github.com/KochetovNicolai))
|
||||||
|
* Исправлена компиляция вычисления агрегатных функций для ключа `LowCardinality` (для случая, когда включена настройка `compile`). [#3886](https://github.com/yandex/ClickHouse/pull/3886) ([KochetovNicolai](https://github.com/KochetovNicolai))
|
||||||
|
* Исправлена передача пользователя и пароля для запросов с реплик. [#3957](https://github.com/yandex/ClickHouse/pull/3957) ([alesapin](https://github.com/alesapin)) ([小路](https://github.com/nicelulu))
|
||||||
|
* Исправлен очень редкий race condition возникающий при перечислении таблиц из базы данных типа `Dictionary` во время перезагрузки словарей. [#3970](https://github.com/yandex/ClickHouse/pull/3970) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Исправлен неверный результат в случае использования HAVING с ROLLUP или CUBE. [#3756](https://github.com/yandex/ClickHouse/issues/3756) [#3837](https://github.com/yandex/ClickHouse/pull/3837) ([Sam Chou](https://github.com/reflection))
|
||||||
|
* Исправлена проблема с алиасами столбцов для запросов с `JOIN ON` над распределенными таблицами. [#3980](https://github.com/yandex/ClickHouse/pull/3980) ([Winter Zhang](https://github.com/zhang2014))
|
||||||
|
* Исправлена ошибка в реализации функции `quantileTDigest` (нашел Artem Vakhrushev). Эта ошибка никогда не происходит в ClickHouse и актуальна только для тех, кто использует кодовую базу ClickHouse напрямую в качестве библиотеки. [#3935](https://github.com/yandex/ClickHouse/pull/3935) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
|
||||||
|
### Улучшения:
|
||||||
|
|
||||||
|
* Добавлена поддержка `IF NOT EXISTS` в выражении `ALTER TABLE ADD COLUMN`, `IF EXISTS` в выражении `DROP/MODIFY/CLEAR/COMMENT COLUMN`. [#3900](https://github.com/yandex/ClickHouse/pull/3900) ([Boris Granveaud](https://github.com/bgranvea))
|
||||||
|
* Функция `parseDateTimeBestEffort` теперь поддерживает форматы `DD.MM.YYYY`, `DD.MM.YY`, `DD-MM-YYYY`, `DD-Mon-YYYY`, `DD/Month/YYYY` и аналогичные. [#3922](https://github.com/yandex/ClickHouse/pull/3922) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* `CapnProtoInputStream` теперь поддерживает jagged структуры. [#4063](https://github.com/yandex/ClickHouse/pull/4063) ([Odin Hultgren Van Der Horst](https://github.com/Miniwoffer))
|
||||||
|
* Улучшение usability: добавлена проверка, что сервер запущен от пользователя, совпадающего с владельцем директории данных. Запрещен запуск от пользователя root в случае, если root не владеет директорией с данными. [#3785](https://github.com/yandex/ClickHouse/pull/3785) ([sergey-v-galtsev](https://github.com/sergey-v-galtsev))
|
||||||
|
* Улучшена логика проверки столбцов, необходимых для JOIN, на стадии анализа запроса. [#3930](https://github.com/yandex/ClickHouse/pull/3930) ([Artem Zuikov](https://github.com/4ertus2))
|
||||||
|
* Уменьшено число поддерживаемых соединений в случае большого числа распределенных таблиц. [#3726](https://github.com/yandex/ClickHouse/pull/3726) ([Winter Zhang](https://github.com/zhang2014))
|
||||||
|
* Добавлена поддержка строки с totals для запроса с `WITH TOTALS` через ODBC драйвер. [#3836](https://github.com/yandex/ClickHouse/pull/3836) ([Maksim Koritckiy](https://github.com/nightweb))
|
||||||
|
* Поддержано использование `Enum` в качестве чисел в функции `if`. [#3875](https://github.com/yandex/ClickHouse/pull/3875) ([Ivan](https://github.com/abyss7))
|
||||||
|
* Добавлена настройка `low_cardinality_allow_in_native_format`. Если она выключена, то тип `LowCadrinality` не используется в формате `Native`. [#3879](https://github.com/yandex/ClickHouse/pull/3879) ([KochetovNicolai](https://github.com/KochetovNicolai))
|
||||||
|
* Удалены некоторые избыточные объекты из кеша скомпилированных выражений для уменьшения потребления памяти. [#4042](https://github.com/yandex/ClickHouse/pull/4042) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Добавлена проверка того, что в запрос `SET send_logs_level = 'value'` передается верное значение. [#3873](https://github.com/yandex/ClickHouse/pull/3873) ([Sabyanin Maxim](https://github.com/s-mx))
|
||||||
|
* Добавлена проверка типов для функций преобразования типов. [#3896](https://github.com/yandex/ClickHouse/pull/3896) ([Winter Zhang](https://github.com/zhang2014))
|
||||||
|
|
||||||
|
### Улучшения производительности:
|
||||||
|
|
||||||
|
* Добавлена настройка `use_minimalistic_part_header_in_zookeeper` для движка MergeTree. Если настройка включена, Replicated таблицы будут хранить метаданные куска в компактном виде (в соответствующем znode для этого куска). Это может значительно уменьшить размер для ZooKeeper snapshot (особенно для таблиц с большим числом столбцов). После включения данной настройки будет невозможно сделать откат к версии, которая эту настройку не поддерживает. [#3960](https://github.com/yandex/ClickHouse/pull/3960) ([Alex Zatelepin](https://github.com/ztlpn))
|
||||||
|
* Добавлена реализация функций `sequenceMatch` и `sequenceCount` на основе конечного автомата в случае, если последовательность событий не содержит условия на время. [#4004](https://github.com/yandex/ClickHouse/pull/4004) ([Léo Ercolanelli](https://github.com/ercolanelli-leo))
|
||||||
|
* Улучшена производительность сериализации целых чисел. [#3968](https://github.com/yandex/ClickHouse/pull/3968) ([Amos Bird](https://github.com/amosbird))
|
||||||
|
* Добавлен zero left padding для PODArray. Теперь элемент с индексом -1 является валидным нулевым значением. Эта особенность используется для удаления условного выражения при вычислении оффсетов массивов. [#3920](https://github.com/yandex/ClickHouse/pull/3920) ([Amos Bird](https://github.com/amosbird))
|
||||||
|
* Откат версии `jemalloc`, приводящей к деградации производительности. [#4018](https://github.com/yandex/ClickHouse/pull/4018) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
|
||||||
|
### Обратно несовместимые изменения:
|
||||||
|
|
||||||
|
* Удалена недокументированная возможность `ALTER MODIFY PRIMARY KEY`, замененная выражением `ALTER MODIFY ORDER BY`. [#3887](https://github.com/yandex/ClickHouse/pull/3887) ([Alex Zatelepin](https://github.com/ztlpn))
|
||||||
|
* Удалена функция `shardByHash`. [#3833](https://github.com/yandex/ClickHouse/pull/3833) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Запрещено использование скалярных подзапросов с результатом, имеющим тип `AggregateFunction`. [#3865](https://github.com/yandex/ClickHouse/pull/3865) ([Ivan](https://github.com/abyss7))
|
||||||
|
|
||||||
|
### Улучшения сборки/тестирования/пакетирования:
|
||||||
|
|
||||||
|
* Добавлена поддержка сборки под PowerPC (`ppc64le`). [#4132](https://github.com/yandex/ClickHouse/pull/4132) ([Danila Kutenin](https://github.com/danlark1))
|
||||||
|
* Функциональные stateful тесты запускаются на публично доступных данных. [#3969](https://github.com/yandex/ClickHouse/pull/3969) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Исправлена ошибка, при которой сервер не мог запуститься с сообщением `bash: /usr/bin/clickhouse-extract-from-config: Operation not permitted` при использовании Docker или systemd-nspawn. [#4136](https://github.com/yandex/ClickHouse/pull/4136) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Обновлена библиотека `rdkafka` до версии v1.0.0-RC5. Использована cppkafka на замену интерфейса языка C. [#4025](https://github.com/yandex/ClickHouse/pull/4025) ([Ivan](https://github.com/abyss7))
|
||||||
|
* Обновлена библиотека `mariadb-client`. Исправлена проблема, обнаруженная с использованием UBSan. [#3924](https://github.com/yandex/ClickHouse/pull/3924) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Исправления для сборок с UBSan. [#3926](https://github.com/yandex/ClickHouse/pull/3926) [#3021](https://github.com/yandex/ClickHouse/pull/3021) [#3948](https://github.com/yandex/ClickHouse/pull/3948) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Добавлены покоммитные запуски тестов с UBSan сборкой.
|
||||||
|
* Добавлены покоммитные запуски тестов со статическим анализатором PVS-Studio.
|
||||||
|
* Исправлены проблемы, найденные с использованием PVS-Studio. [#4013](https://github.com/yandex/ClickHouse/pull/4013) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Исправлены проблемы совместимости glibc. [#4100](https://github.com/yandex/ClickHouse/pull/4100) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Docker образы перемещены на Ubuntu 18.10, добавлена совместимость с glibc >= 2.28 [#3965](https://github.com/yandex/ClickHouse/pull/3965) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Добавлена переменная окружения `CLICKHOUSE_DO_NOT_CHOWN`, позволяющая не делать shown директории для Docker образа сервера. [#3967](https://github.com/yandex/ClickHouse/pull/3967) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Включены большинство предупреждений из `-Weverything` для clang. Включено `-Wpedantic`. [#3986](https://github.com/yandex/ClickHouse/pull/3986) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Добавлены некоторые предупреждения, специфичные только для clang 8. [#3993](https://github.com/yandex/ClickHouse/pull/3993) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* При использовании динамической линковки используется `libLLVM` вместо библиотеки `LLVM`. [#3989](https://github.com/yandex/ClickHouse/pull/3989) ([Orivej Desh](https://github.com/orivej))
|
||||||
|
* Добавлены переменные окружения для параметров `TSan`, `UBSan`, `ASan` в тестовом Docker образе. [#4072](https://github.com/yandex/ClickHouse/pull/4072) ([alesapin](https://github.com/alesapin))
|
||||||
|
* Debian пакет `clickhouse-server` будет рекомендовать пакет `libcap2-bin` для того, чтобы использовать утилиту `setcap` для настроек. Данный пакет опционален. [#4093](https://github.com/yandex/ClickHouse/pull/4093) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Уменьшено время сборки, убраны ненужные включения заголовочных файлов. [#3898](https://github.com/yandex/ClickHouse/pull/3898) ([proller](https://github.com/proller))
|
||||||
|
* Добавлены тесты производительности для функций хеширования. [#3918](https://github.com/yandex/ClickHouse/pull/3918) ([filimonov](https://github.com/filimonov))
|
||||||
|
* Исправлены циклические зависимости библиотек. [#3958](https://github.com/yandex/ClickHouse/pull/3958) ([proller](https://github.com/proller))
|
||||||
|
* Улучшена компиляция при малом объеме памяти. [#4030](https://github.com/yandex/ClickHouse/pull/4030) ([proller](https://github.com/proller))
|
||||||
|
* Добавлен тестовый скрипт для воспроизведения деградации производительности в `jemalloc`. [#4036](https://github.com/yandex/ClickHouse/pull/4036) ([alexey-milovidov](https://github.com/alexey-milovidov))
|
||||||
|
* Исправления опечаток в комментариях и строковых литералах. [#4122](https://github.com/yandex/ClickHouse/pull/4122) ([maiha](https://github.com/maiha))
|
||||||
|
* Исправления опечаток в комментариях. [#4089](https://github.com/yandex/ClickHouse/pull/4089) ([Evgenii Pravda](https://github.com/kvinty))
|
||||||
|
|
||||||
## ClickHouse release 18.16.1, 2018-12-21
|
## ClickHouse release 18.16.1, 2018-12-21
|
||||||
|
|
||||||
### Исправления ошибок:
|
### Исправления ошибок:
|
||||||
|
|||||||
143
CMakeLists.txt
143
CMakeLists.txt
@ -3,6 +3,21 @@ cmake_minimum_required (VERSION 3.3)
|
|||||||
|
|
||||||
set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} "${CMAKE_CURRENT_SOURCE_DIR}/cmake/Modules/")
|
set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} "${CMAKE_CURRENT_SOURCE_DIR}/cmake/Modules/")
|
||||||
|
|
||||||
|
option(ENABLE_IPO "Enable inter-procedural optimization (aka LTO)" OFF) # need cmake 3.9+
|
||||||
|
if(ENABLE_IPO)
|
||||||
|
cmake_policy(SET CMP0069 NEW)
|
||||||
|
include(CheckIPOSupported)
|
||||||
|
check_ipo_supported(RESULT IPO_SUPPORTED OUTPUT IPO_NOT_SUPPORTED)
|
||||||
|
if(IPO_SUPPORTED)
|
||||||
|
message(STATUS "IPO/LTO is supported, enabling")
|
||||||
|
set(CMAKE_INTERPROCEDURAL_OPTIMIZATION TRUE)
|
||||||
|
else()
|
||||||
|
message(STATUS "IPO/LTO is not supported: <${IPO_NOT_SUPPORTED}>")
|
||||||
|
endif()
|
||||||
|
else()
|
||||||
|
message(STATUS "IPO/LTO not enabled.")
|
||||||
|
endif()
|
||||||
|
|
||||||
if (CMAKE_CXX_COMPILER_ID STREQUAL "GNU")
|
if (CMAKE_CXX_COMPILER_ID STREQUAL "GNU")
|
||||||
# Require at least gcc 7
|
# Require at least gcc 7
|
||||||
if (CMAKE_CXX_COMPILER_VERSION VERSION_LESS 7 AND NOT CMAKE_VERSION VERSION_LESS 2.8.9)
|
if (CMAKE_CXX_COMPILER_VERSION VERSION_LESS 7 AND NOT CMAKE_VERSION VERSION_LESS 2.8.9)
|
||||||
@ -35,6 +50,7 @@ string(TOUPPER ${CMAKE_BUILD_TYPE} CMAKE_BUILD_TYPE_UC)
|
|||||||
message (STATUS "CMAKE_BUILD_TYPE: ${CMAKE_BUILD_TYPE}")
|
message (STATUS "CMAKE_BUILD_TYPE: ${CMAKE_BUILD_TYPE}")
|
||||||
|
|
||||||
set (CMAKE_CONFIGURATION_TYPES "RelWithDebInfo;Debug;Release;MinSizeRel" CACHE STRING "" FORCE)
|
set (CMAKE_CONFIGURATION_TYPES "RelWithDebInfo;Debug;Release;MinSizeRel" CACHE STRING "" FORCE)
|
||||||
|
set (CMAKE_DEBUG_POSTFIX "d" CACHE STRING "Generate debug library name with a postfix.") # To be consistent with CMakeLists from contrib libs.
|
||||||
|
|
||||||
option (USE_STATIC_LIBRARIES "Set to FALSE to use shared libraries" ON)
|
option (USE_STATIC_LIBRARIES "Set to FALSE to use shared libraries" ON)
|
||||||
option (MAKE_STATIC_LIBRARIES "Set to FALSE to make shared libraries" ${USE_STATIC_LIBRARIES})
|
option (MAKE_STATIC_LIBRARIES "Set to FALSE to make shared libraries" ${USE_STATIC_LIBRARIES})
|
||||||
@ -81,7 +97,7 @@ option (ENABLE_TESTS "Enables tests" ON)
|
|||||||
if (CMAKE_SYSTEM_PROCESSOR MATCHES "amd64|x86_64")
|
if (CMAKE_SYSTEM_PROCESSOR MATCHES "amd64|x86_64")
|
||||||
option (USE_INTERNAL_MEMCPY "Use internal implementation of 'memcpy' function instead of provided by libc. Only for x86_64." ON)
|
option (USE_INTERNAL_MEMCPY "Use internal implementation of 'memcpy' function instead of provided by libc. Only for x86_64." ON)
|
||||||
|
|
||||||
if (OS_LINUX)
|
if (OS_LINUX AND NOT UNBUNDLED AND MAKE_STATIC_LIBRARIES AND CMAKE_VERSION VERSION_GREATER "3.9.0")
|
||||||
option (GLIBC_COMPATIBILITY "Set to TRUE to enable compatibility with older glibc libraries. Only for x86_64, Linux. Implies USE_INTERNAL_MEMCPY." ON)
|
option (GLIBC_COMPATIBILITY "Set to TRUE to enable compatibility with older glibc libraries. Only for x86_64, Linux. Implies USE_INTERNAL_MEMCPY." ON)
|
||||||
endif ()
|
endif ()
|
||||||
endif ()
|
endif ()
|
||||||
@ -90,8 +106,6 @@ if (GLIBC_COMPATIBILITY)
|
|||||||
set (USE_INTERNAL_MEMCPY ON)
|
set (USE_INTERNAL_MEMCPY ON)
|
||||||
endif ()
|
endif ()
|
||||||
|
|
||||||
set (COMPILER_FLAGS "${COMPILER_FLAGS}")
|
|
||||||
|
|
||||||
string(REGEX MATCH "-?[0-9]+(.[0-9]+)?$" COMPILER_POSTFIX ${CMAKE_CXX_COMPILER})
|
string(REGEX MATCH "-?[0-9]+(.[0-9]+)?$" COMPILER_POSTFIX ${CMAKE_CXX_COMPILER})
|
||||||
|
|
||||||
find_program (LLD_PATH NAMES "lld${COMPILER_POSTFIX}" "lld")
|
find_program (LLD_PATH NAMES "lld${COMPILER_POSTFIX}" "lld")
|
||||||
@ -108,12 +122,19 @@ if (LINKER_NAME)
|
|||||||
set (CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fuse-ld=${LINKER_NAME}")
|
set (CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fuse-ld=${LINKER_NAME}")
|
||||||
endif ()
|
endif ()
|
||||||
|
|
||||||
option (PIPE "-pipe compiler option [less /tmp usage, more ram usage]" ON)
|
cmake_host_system_information(RESULT AVAILABLE_PHYSICAL_MEMORY QUERY AVAILABLE_PHYSICAL_MEMORY) # Not available under freebsd
|
||||||
if (PIPE)
|
if(NOT AVAILABLE_PHYSICAL_MEMORY OR AVAILABLE_PHYSICAL_MEMORY GREATER 8000)
|
||||||
|
option(COMPILER_PIPE "-pipe compiler option [less /tmp usage, more ram usage]" ON)
|
||||||
|
endif()
|
||||||
|
if(COMPILER_PIPE)
|
||||||
set(COMPILER_FLAGS "${COMPILER_FLAGS} -pipe")
|
set(COMPILER_FLAGS "${COMPILER_FLAGS} -pipe")
|
||||||
|
else()
|
||||||
|
message(STATUS "Disabling compiler -pipe option (have only ${AVAILABLE_PHYSICAL_MEMORY} mb of memory)")
|
||||||
endif()
|
endif()
|
||||||
|
|
||||||
|
if(NOT DISABLE_CPU_OPTIMIZE)
|
||||||
include(cmake/test_cpu.cmake)
|
include(cmake/test_cpu.cmake)
|
||||||
|
endif()
|
||||||
|
|
||||||
if(NOT COMPILER_CLANG) # clang: error: the clang compiler does not support '-march=native'
|
if(NOT COMPILER_CLANG) # clang: error: the clang compiler does not support '-march=native'
|
||||||
option(ARCH_NATIVE "Enable -march=native compiler flag" ${ARCH_ARM})
|
option(ARCH_NATIVE "Enable -march=native compiler flag" ${ARCH_ARM})
|
||||||
@ -190,6 +211,60 @@ if (USE_STATIC_LIBRARIES AND HAVE_NO_PIE)
|
|||||||
endif ()
|
endif ()
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
# Set standard, system and compiler libraries explicitly.
|
||||||
|
# This is intended for more control of what we are linking.
|
||||||
|
|
||||||
|
set (DEFAULT_LIBS "")
|
||||||
|
if (OS_LINUX AND NOT UNBUNDLED)
|
||||||
|
# Note: this probably has no effict, but I'm not an expert in CMake.
|
||||||
|
set (CMAKE_C_IMPLICIT_LINK_LIBRARIES "")
|
||||||
|
set (CMAKE_CXX_IMPLICIT_LINK_LIBRARIES "")
|
||||||
|
|
||||||
|
# Disable default linked libraries.
|
||||||
|
set (DEFAULT_LIBS "-nodefaultlibs")
|
||||||
|
|
||||||
|
# Add C++ libraries.
|
||||||
|
#
|
||||||
|
# This consist of:
|
||||||
|
# - C++ standard library (like implementation of std::string);
|
||||||
|
# - C++ ABI implementation (functions for exceptions like __cxa_throw, RTTI, etc);
|
||||||
|
# - functions for internal implementation of exception handling (stack unwinding based on DWARF info; TODO replace with bundled libunwind);
|
||||||
|
# - compiler builtins (example: functions for implementation of __int128 operations);
|
||||||
|
#
|
||||||
|
# There are two variants of C++ library: libc++ (from LLVM compiler infrastructure) and libstdc++ (from GCC).
|
||||||
|
if (USE_LIBCXX)
|
||||||
|
set (BUILTINS_LIB_PATH "")
|
||||||
|
if (COMPILER_CLANG)
|
||||||
|
execute_process (COMMAND ${CMAKE_CXX_COMPILER} --print-file-name=libclang_rt.builtins-${CMAKE_SYSTEM_PROCESSOR}.a OUTPUT_VARIABLE BUILTINS_LIB_PATH OUTPUT_STRIP_TRAILING_WHITESPACE)
|
||||||
|
endif ()
|
||||||
|
|
||||||
|
set (DEFAULT_LIBS "${DEFAULT_LIBS} -Wl,-Bstatic ${LIBCXX_LIBRARY} ${LIBCXXABI_LIBRARY} -lgcc_eh ${BUILTINS_LIB_PATH} -Wl,-Bdynamic")
|
||||||
|
else ()
|
||||||
|
set (DEFAULT_LIBS "${DEFAULT_LIBS} -Wl,-Bstatic -lstdc++ -lgcc_eh -lgcc -Wl,-Bdynamic")
|
||||||
|
endif ()
|
||||||
|
|
||||||
|
# Linking with GLIBC prevents portability of binaries to older systems.
|
||||||
|
# We overcome this behaviour by statically linking with our own implementation of all new symbols (that don't exist in older Libc or have infamous "symbol versioning").
|
||||||
|
# The order of linking is important: 'glibc-compatibility' must be before libc but after all other libraries.
|
||||||
|
if (GLIBC_COMPATIBILITY)
|
||||||
|
message (STATUS "Some symbols from glibc will be replaced for compatibility")
|
||||||
|
|
||||||
|
string (TOUPPER ${CMAKE_BUILD_TYPE} CMAKE_BUILD_TYPE_UC)
|
||||||
|
set (CMAKE_POSTFIX_VARIABLE "CMAKE_${CMAKE_BUILD_TYPE_UC}_POSTFIX")
|
||||||
|
|
||||||
|
set (DEFAULT_LIBS "${DEFAULT_LIBS} libs/libglibc-compatibility/libglibc-compatibility${${CMAKE_POSTFIX_VARIABLE}}.a")
|
||||||
|
endif ()
|
||||||
|
|
||||||
|
# Add Libc. GLIBC is actually a collection of interdependent libraries.
|
||||||
|
set (DEFAULT_LIBS "${DEFAULT_LIBS} -lrt -ldl -lpthread -lm -lc")
|
||||||
|
|
||||||
|
# Note: we'd rather use Musl libc library, but it's little bit more difficult to use.
|
||||||
|
|
||||||
|
message(STATUS "Default libraries: ${DEFAULT_LIBS}")
|
||||||
|
endif ()
|
||||||
|
|
||||||
|
|
||||||
if (NOT MAKE_STATIC_LIBRARIES)
|
if (NOT MAKE_STATIC_LIBRARIES)
|
||||||
set(CMAKE_POSITION_INDEPENDENT_CODE ON)
|
set(CMAKE_POSITION_INDEPENDENT_CODE ON)
|
||||||
endif ()
|
endif ()
|
||||||
@ -213,8 +288,8 @@ endif (TEST_COVERAGE)
|
|||||||
|
|
||||||
if (ENABLE_TESTS)
|
if (ENABLE_TESTS)
|
||||||
message (STATUS "Tests are enabled")
|
message (STATUS "Tests are enabled")
|
||||||
enable_testing()
|
|
||||||
endif ()
|
endif ()
|
||||||
|
enable_testing() # Enable for tests without binary
|
||||||
|
|
||||||
# when installing to /usr - place configs to /etc but for /usr/local place to /usr/local/etc
|
# when installing to /usr - place configs to /etc but for /usr/local place to /usr/local/etc
|
||||||
if (CMAKE_INSTALL_PREFIX STREQUAL "/usr")
|
if (CMAKE_INSTALL_PREFIX STREQUAL "/usr")
|
||||||
@ -234,9 +309,10 @@ if (UNBUNDLED OR NOT (OS_LINUX OR APPLE) OR ARCH_32)
|
|||||||
option (NO_WERROR "Disable -Werror compiler option" ON)
|
option (NO_WERROR "Disable -Werror compiler option" ON)
|
||||||
endif ()
|
endif ()
|
||||||
|
|
||||||
message (STATUS "Building for: ${CMAKE_SYSTEM} ${CMAKE_SYSTEM_PROCESSOR} ${CMAKE_LIBRARY_ARCHITECTURE} ; USE_STATIC_LIBRARIES=${USE_STATIC_LIBRARIES} MAKE_STATIC_LIBRARIES=${MAKE_STATIC_LIBRARIES} UNBUNDLED=${UNBUNDLED} CCACHE=${CCACHE_FOUND} ${CCACHE_VERSION}")
|
message (STATUS "Building for: ${CMAKE_SYSTEM} ${CMAKE_SYSTEM_PROCESSOR} ${CMAKE_LIBRARY_ARCHITECTURE} ; USE_STATIC_LIBRARIES=${USE_STATIC_LIBRARIES} MAKE_STATIC_LIBRARIES=${MAKE_STATIC_LIBRARIES} SPLIT_SHARED=${SPLIT_SHARED_LIBRARIES} UNBUNDLED=${UNBUNDLED} CCACHE=${CCACHE_FOUND} ${CCACHE_VERSION}")
|
||||||
|
|
||||||
include(GNUInstallDirs)
|
include(GNUInstallDirs)
|
||||||
|
include (cmake/find_contrib_lib.cmake)
|
||||||
|
|
||||||
include (cmake/find_ssl.cmake)
|
include (cmake/find_ssl.cmake)
|
||||||
include (cmake/lib_name.cmake)
|
include (cmake/lib_name.cmake)
|
||||||
@ -259,25 +335,34 @@ include (cmake/find_re2.cmake)
|
|||||||
include (cmake/find_rdkafka.cmake)
|
include (cmake/find_rdkafka.cmake)
|
||||||
include (cmake/find_capnp.cmake)
|
include (cmake/find_capnp.cmake)
|
||||||
include (cmake/find_llvm.cmake)
|
include (cmake/find_llvm.cmake)
|
||||||
include (cmake/find_cpuid.cmake)
|
include (cmake/find_cpuid.cmake) # Freebsd, bundled
|
||||||
|
if (NOT USE_CPUID)
|
||||||
|
include (cmake/find_cpuinfo.cmake) # Debian
|
||||||
|
endif()
|
||||||
include (cmake/find_libgsasl.cmake)
|
include (cmake/find_libgsasl.cmake)
|
||||||
include (cmake/find_libxml2.cmake)
|
include (cmake/find_libxml2.cmake)
|
||||||
include (cmake/find_hdfs3.cmake)
|
include (cmake/find_brotli.cmake)
|
||||||
|
include (cmake/find_protobuf.cmake)
|
||||||
|
include (cmake/find_pdqsort.cmake)
|
||||||
|
include (cmake/find_hdfs3.cmake) # uses protobuf
|
||||||
include (cmake/find_consistent-hashing.cmake)
|
include (cmake/find_consistent-hashing.cmake)
|
||||||
include (cmake/find_base64.cmake)
|
include (cmake/find_base64.cmake)
|
||||||
include (cmake/find_cxxabi.cmake)
|
include (cmake/find_cxxabi.cmake)
|
||||||
include (cmake/find_cxx.cmake)
|
include (cmake/find_cxx.cmake)
|
||||||
if (ENABLE_TESTS)
|
include (cmake/find_parquet.cmake)
|
||||||
include (cmake/find_gtest.cmake)
|
|
||||||
endif ()
|
|
||||||
|
|
||||||
include (cmake/find_contrib_lib.cmake)
|
include (cmake/find_contrib_lib.cmake)
|
||||||
|
|
||||||
find_contrib_lib(cityhash)
|
find_contrib_lib(cityhash)
|
||||||
find_contrib_lib(farmhash)
|
find_contrib_lib(farmhash)
|
||||||
find_contrib_lib(metrohash)
|
find_contrib_lib(metrohash)
|
||||||
find_contrib_lib(btrie)
|
find_contrib_lib(btrie)
|
||||||
find_contrib_lib(double-conversion)
|
find_contrib_lib(double-conversion)
|
||||||
|
|
||||||
|
if (ENABLE_TESTS)
|
||||||
|
include (cmake/find_gtest.cmake)
|
||||||
|
endif ()
|
||||||
|
|
||||||
if (USE_INTERNAL_LIBCXX_LIBRARY)
|
if (USE_INTERNAL_LIBCXX_LIBRARY)
|
||||||
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -nostdinc++ -isystem ${LIBCXX_INCLUDE_DIR} -isystem ${LIBCXXABI_INCLUDE_DIR}")
|
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -nostdinc++ -isystem ${LIBCXX_INCLUDE_DIR} -isystem ${LIBCXXABI_INCLUDE_DIR}")
|
||||||
endif ()
|
endif ()
|
||||||
@ -291,9 +376,41 @@ include (libs/libdaemon/cmake/find_unwind.cmake)
|
|||||||
|
|
||||||
include (cmake/print_flags.cmake)
|
include (cmake/print_flags.cmake)
|
||||||
|
|
||||||
add_subdirectory (contrib)
|
add_subdirectory (contrib EXCLUDE_FROM_ALL)
|
||||||
add_subdirectory (libs)
|
add_subdirectory (libs)
|
||||||
add_subdirectory (utils)
|
add_subdirectory (utils)
|
||||||
add_subdirectory (dbms)
|
add_subdirectory (dbms)
|
||||||
|
|
||||||
include (cmake/print_include_directories.cmake)
|
include (cmake/print_include_directories.cmake)
|
||||||
|
|
||||||
|
|
||||||
|
if (DEFAULT_LIBS)
|
||||||
|
# Add default libs to all targets as the last dependency.
|
||||||
|
# I have found no better way to specify default libs in CMake that will appear single time in specific order at the end of linker arguments.
|
||||||
|
|
||||||
|
function(add_default_libs target_name)
|
||||||
|
if (TARGET ${target_name})
|
||||||
|
# message(STATUS "Has target ${target_name}")
|
||||||
|
set_property(TARGET ${target_name} APPEND PROPERTY LINK_LIBRARIES "${DEFAULT_LIBS}")
|
||||||
|
set_property(TARGET ${target_name} APPEND PROPERTY INTERFACE_LINK_LIBRARIES "${DEFAULT_LIBS}")
|
||||||
|
if (GLIBC_COMPATIBILITY)
|
||||||
|
add_dependencies(${target_name} glibc-compatibility)
|
||||||
|
endif ()
|
||||||
|
endif ()
|
||||||
|
endfunction ()
|
||||||
|
|
||||||
|
add_default_libs(ltdl)
|
||||||
|
add_default_libs(zlibstatic)
|
||||||
|
add_default_libs(jemalloc)
|
||||||
|
add_default_libs(unwind)
|
||||||
|
add_default_libs(memcpy)
|
||||||
|
add_default_libs(Foundation)
|
||||||
|
add_default_libs(common)
|
||||||
|
add_default_libs(gtest)
|
||||||
|
add_default_libs(lz4)
|
||||||
|
add_default_libs(zstd)
|
||||||
|
add_default_libs(snappy)
|
||||||
|
add_default_libs(arrow)
|
||||||
|
add_default_libs(thrift_static)
|
||||||
|
add_default_libs(boost_regex_internal)
|
||||||
|
endif ()
|
||||||
|
|||||||
2
LICENSE
2
LICENSE
@ -186,7 +186,7 @@
|
|||||||
same "printed page" as the copyright notice for easier
|
same "printed page" as the copyright notice for easier
|
||||||
identification within third-party archives.
|
identification within third-party archives.
|
||||||
|
|
||||||
Copyright 2016-2018 Yandex LLC
|
Copyright 2016-2019 Yandex LLC
|
||||||
|
|
||||||
Licensed under the Apache License, Version 2.0 (the "License");
|
Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
you may not use this file except in compliance with the License.
|
you may not use this file except in compliance with the License.
|
||||||
|
|||||||
@ -1,7 +1,7 @@
|
|||||||
# Use Ninja instead of Unix Makefiles by default.
|
# Use Ninja instead of Unix Makefiles by default.
|
||||||
# https://stackoverflow.com/questions/11269833/cmake-selecting-a-generator-within-cmakelists-txt
|
# https://stackoverflow.com/questions/11269833/cmake-selecting-a-generator-within-cmakelists-txt
|
||||||
#
|
#
|
||||||
# Reason: it have better startup time than make and it parallelize jobs more uniformly.
|
# Reason: it has better startup time than make and it parallelizes jobs more uniformly.
|
||||||
# (when comparing to make with Makefiles that was generated by CMake)
|
# (when comparing to make with Makefiles that was generated by CMake)
|
||||||
#
|
#
|
||||||
# How to install Ninja on Ubuntu:
|
# How to install Ninja on Ubuntu:
|
||||||
|
|||||||
@ -10,3 +10,8 @@ ClickHouse is an open-source column-oriented database management system that all
|
|||||||
* [Blog](https://clickhouse.yandex/blog/en/) contains various ClickHouse-related articles, as well as announces and reports about events.
|
* [Blog](https://clickhouse.yandex/blog/en/) contains various ClickHouse-related articles, as well as announces and reports about events.
|
||||||
* [Contacts](https://clickhouse.yandex/#contacts) can help to get your questions answered if there are any.
|
* [Contacts](https://clickhouse.yandex/#contacts) can help to get your questions answered if there are any.
|
||||||
* You can also [fill this form](https://forms.yandex.com/surveys/meet-yandex-clickhouse-team/) to meet Yandex ClickHouse team in person.
|
* You can also [fill this form](https://forms.yandex.com/surveys/meet-yandex-clickhouse-team/) to meet Yandex ClickHouse team in person.
|
||||||
|
|
||||||
|
## Upcoming Events
|
||||||
|
|
||||||
|
* [ClickHouse Community Meetup](https://www.eventbrite.com/e/meetup-clickhouse-in-the-wild-deployment-success-stories-registration-55305051899) in San Francisco on February 19.
|
||||||
|
* [ClickHouse Community Meetup](https://www.eventbrite.com/e/clickhouse-meetup-in-madrid-registration-55376746339) in Madrid on April 2.
|
||||||
|
|||||||
@ -25,9 +25,10 @@ Various possible options. We are not going to automate testing all of them.
|
|||||||
|
|
||||||
#### CPU architectures:
|
#### CPU architectures:
|
||||||
- x86_64;
|
- x86_64;
|
||||||
- AArch64.
|
- AArch64;
|
||||||
|
- PowerPC64LE.
|
||||||
|
|
||||||
x86_64 is the main CPU architecture. We also have minimal support for AArch64.
|
x86_64 is the main CPU architecture. We also have minimal support for AArch64 and PowerPC64LE.
|
||||||
|
|
||||||
#### Operating systems:
|
#### Operating systems:
|
||||||
- Linux;
|
- Linux;
|
||||||
|
|||||||
@ -3,7 +3,6 @@ set -e -x
|
|||||||
|
|
||||||
source default-config
|
source default-config
|
||||||
|
|
||||||
# TODO Non debian systems
|
|
||||||
./install-os-packages.sh svn
|
./install-os-packages.sh svn
|
||||||
./install-os-packages.sh cmake
|
./install-os-packages.sh cmake
|
||||||
|
|
||||||
|
|||||||
159
cmake/Modules/FindArrow.cmake
Normal file
159
cmake/Modules/FindArrow.cmake
Normal file
@ -0,0 +1,159 @@
|
|||||||
|
# https://github.com/apache/arrow/blob/master/cpp/cmake_modules/FindArrow.cmake
|
||||||
|
|
||||||
|
# Licensed to the Apache Software Foundation (ASF) under one
|
||||||
|
# or more contributor license agreements. See the NOTICE file
|
||||||
|
# distributed with this work for additional information
|
||||||
|
# regarding copyright ownership. The ASF licenses this file
|
||||||
|
# to you under the Apache License, Version 2.0 (the
|
||||||
|
# "License"); you may not use this file except in compliance
|
||||||
|
# with the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing,
|
||||||
|
# software distributed under the License is distributed on an
|
||||||
|
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||||
|
# KIND, either express or implied. See the License for the
|
||||||
|
# specific language governing permissions and limitations
|
||||||
|
# under the License.
|
||||||
|
|
||||||
|
# - Find ARROW (arrow/api.h, libarrow.a, libarrow.so)
|
||||||
|
# This module defines
|
||||||
|
# ARROW_INCLUDE_DIR, directory containing headers
|
||||||
|
# ARROW_LIBS, directory containing arrow libraries
|
||||||
|
# ARROW_STATIC_LIB, path to libarrow.a
|
||||||
|
# ARROW_SHARED_LIB, path to libarrow's shared library
|
||||||
|
# ARROW_SHARED_IMP_LIB, path to libarrow's import library (MSVC only)
|
||||||
|
# ARROW_FOUND, whether arrow has been found
|
||||||
|
|
||||||
|
include(FindPkgConfig)
|
||||||
|
include(GNUInstallDirs)
|
||||||
|
|
||||||
|
if ("$ENV{ARROW_HOME}" STREQUAL "")
|
||||||
|
pkg_check_modules(ARROW arrow)
|
||||||
|
if (ARROW_FOUND)
|
||||||
|
pkg_get_variable(ARROW_SO_VERSION arrow so_version)
|
||||||
|
set(ARROW_ABI_VERSION ${ARROW_SO_VERSION})
|
||||||
|
message(STATUS "Arrow SO and ABI version: ${ARROW_SO_VERSION}")
|
||||||
|
pkg_get_variable(ARROW_FULL_SO_VERSION arrow full_so_version)
|
||||||
|
message(STATUS "Arrow full SO version: ${ARROW_FULL_SO_VERSION}")
|
||||||
|
if ("${ARROW_INCLUDE_DIRS}" STREQUAL "")
|
||||||
|
set(ARROW_INCLUDE_DIRS "/usr/${CMAKE_INSTALL_INCLUDEDIR}")
|
||||||
|
endif()
|
||||||
|
if ("${ARROW_LIBRARY_DIRS}" STREQUAL "")
|
||||||
|
set(ARROW_LIBRARY_DIRS "/usr/${CMAKE_INSTALL_LIBDIR}")
|
||||||
|
if (EXISTS "/etc/debian_version" AND CMAKE_LIBRARY_ARCHITECTURE)
|
||||||
|
set(ARROW_LIBRARY_DIRS
|
||||||
|
"${ARROW_LIBRARY_DIRS}/${CMAKE_LIBRARY_ARCHITECTURE}")
|
||||||
|
endif()
|
||||||
|
endif()
|
||||||
|
set(ARROW_INCLUDE_DIR ${ARROW_INCLUDE_DIRS})
|
||||||
|
set(ARROW_LIBS ${ARROW_LIBRARY_DIRS})
|
||||||
|
set(ARROW_SEARCH_LIB_PATH ${ARROW_LIBRARY_DIRS})
|
||||||
|
endif()
|
||||||
|
else()
|
||||||
|
set(ARROW_HOME "$ENV{ARROW_HOME}")
|
||||||
|
|
||||||
|
set(ARROW_SEARCH_HEADER_PATHS
|
||||||
|
${ARROW_HOME}/include
|
||||||
|
)
|
||||||
|
|
||||||
|
set(ARROW_SEARCH_LIB_PATH
|
||||||
|
${ARROW_HOME}/lib
|
||||||
|
)
|
||||||
|
|
||||||
|
find_path(ARROW_INCLUDE_DIR arrow/array.h PATHS
|
||||||
|
${ARROW_SEARCH_HEADER_PATHS}
|
||||||
|
# make sure we don't accidentally pick up a different version
|
||||||
|
NO_DEFAULT_PATH
|
||||||
|
)
|
||||||
|
endif()
|
||||||
|
|
||||||
|
find_library(ARROW_LIB_PATH NAMES arrow
|
||||||
|
PATHS
|
||||||
|
${ARROW_SEARCH_LIB_PATH}
|
||||||
|
NO_DEFAULT_PATH)
|
||||||
|
get_filename_component(ARROW_LIBS ${ARROW_LIB_PATH} DIRECTORY)
|
||||||
|
|
||||||
|
find_library(ARROW_PYTHON_LIB_PATH NAMES arrow_python
|
||||||
|
PATHS
|
||||||
|
${ARROW_SEARCH_LIB_PATH}
|
||||||
|
NO_DEFAULT_PATH)
|
||||||
|
get_filename_component(ARROW_PYTHON_LIBS ${ARROW_PYTHON_LIB_PATH} DIRECTORY)
|
||||||
|
|
||||||
|
if (MSVC)
|
||||||
|
SET(CMAKE_FIND_LIBRARY_SUFFIXES ".lib" ".dll")
|
||||||
|
|
||||||
|
if (MSVC AND NOT DEFINED ARROW_MSVC_STATIC_LIB_SUFFIX)
|
||||||
|
set(ARROW_MSVC_STATIC_LIB_SUFFIX "_static")
|
||||||
|
endif()
|
||||||
|
|
||||||
|
find_library(ARROW_SHARED_LIBRARIES NAMES arrow
|
||||||
|
PATHS ${ARROW_HOME} NO_DEFAULT_PATH
|
||||||
|
PATH_SUFFIXES "bin" )
|
||||||
|
|
||||||
|
find_library(ARROW_PYTHON_SHARED_LIBRARIES NAMES arrow_python
|
||||||
|
PATHS ${ARROW_HOME} NO_DEFAULT_PATH
|
||||||
|
PATH_SUFFIXES "bin" )
|
||||||
|
get_filename_component(ARROW_SHARED_LIBS ${ARROW_SHARED_LIBRARIES} PATH )
|
||||||
|
get_filename_component(ARROW_PYTHON_SHARED_LIBS ${ARROW_PYTHON_SHARED_LIBRARIES} PATH )
|
||||||
|
endif ()
|
||||||
|
|
||||||
|
if (ARROW_INCLUDE_DIR AND ARROW_LIBS)
|
||||||
|
set(ARROW_FOUND TRUE)
|
||||||
|
set(ARROW_LIB_NAME arrow)
|
||||||
|
set(ARROW_PYTHON_LIB_NAME arrow_python)
|
||||||
|
if (MSVC)
|
||||||
|
set(ARROW_STATIC_LIB ${ARROW_LIBS}/${ARROW_LIB_NAME}${ARROW_MSVC_STATIC_LIB_SUFFIX}${CMAKE_STATIC_LIBRARY_SUFFIX})
|
||||||
|
set(ARROW_PYTHON_STATIC_LIB ${ARROW_PYTHON_LIBS}/${ARROW_PYTHON_LIB_NAME}${ARROW_MSVC_STATIC_LIB_SUFFIX}${CMAKE_STATIC_LIBRARY_SUFFIX})
|
||||||
|
set(ARROW_SHARED_LIB ${ARROW_SHARED_LIBS}/${ARROW_LIB_NAME}${CMAKE_SHARED_LIBRARY_SUFFIX})
|
||||||
|
set(ARROW_PYTHON_SHARED_LIB ${ARROW_PYTHON_SHARED_LIBS}/${ARROW_PYTHON_LIB_NAME}${CMAKE_SHARED_LIBRARY_SUFFIX})
|
||||||
|
set(ARROW_SHARED_IMP_LIB ${ARROW_LIBS}/${ARROW_LIB_NAME}.lib)
|
||||||
|
set(ARROW_PYTHON_SHARED_IMP_LIB ${ARROW_PYTHON_LIBS}/${ARROW_PYTHON_LIB_NAME}.lib)
|
||||||
|
else()
|
||||||
|
set(ARROW_STATIC_LIB ${ARROW_LIBS}/lib${ARROW_LIB_NAME}.a)
|
||||||
|
set(ARROW_PYTHON_STATIC_LIB ${ARROW_LIBS}/lib${ARROW_PYTHON_LIB_NAME}.a)
|
||||||
|
|
||||||
|
set(ARROW_SHARED_LIB ${ARROW_LIBS}/lib${ARROW_LIB_NAME}${CMAKE_SHARED_LIBRARY_SUFFIX})
|
||||||
|
set(ARROW_PYTHON_SHARED_LIB ${ARROW_LIBS}/lib${ARROW_PYTHON_LIB_NAME}${CMAKE_SHARED_LIBRARY_SUFFIX})
|
||||||
|
endif()
|
||||||
|
endif()
|
||||||
|
|
||||||
|
if (ARROW_FOUND)
|
||||||
|
if (NOT Arrow_FIND_QUIETLY)
|
||||||
|
message(STATUS "Found the Arrow core library: ${ARROW_LIB_PATH}")
|
||||||
|
message(STATUS "Found the Arrow Python library: ${ARROW_PYTHON_LIB_PATH}")
|
||||||
|
endif ()
|
||||||
|
else ()
|
||||||
|
if (NOT Arrow_FIND_QUIETLY)
|
||||||
|
set(ARROW_ERR_MSG "Could not find the Arrow library. Looked for headers")
|
||||||
|
set(ARROW_ERR_MSG "${ARROW_ERR_MSG} in ${ARROW_SEARCH_HEADER_PATHS}, and for libs")
|
||||||
|
set(ARROW_ERR_MSG "${ARROW_ERR_MSG} in ${ARROW_SEARCH_LIB_PATH}")
|
||||||
|
if (Arrow_FIND_REQUIRED)
|
||||||
|
message(FATAL_ERROR "${ARROW_ERR_MSG}")
|
||||||
|
else (Arrow_FIND_REQUIRED)
|
||||||
|
message(STATUS "${ARROW_ERR_MSG}")
|
||||||
|
endif (Arrow_FIND_REQUIRED)
|
||||||
|
endif ()
|
||||||
|
set(ARROW_FOUND FALSE)
|
||||||
|
endif ()
|
||||||
|
|
||||||
|
if (MSVC)
|
||||||
|
mark_as_advanced(
|
||||||
|
ARROW_INCLUDE_DIR
|
||||||
|
ARROW_STATIC_LIB
|
||||||
|
ARROW_SHARED_LIB
|
||||||
|
ARROW_SHARED_IMP_LIB
|
||||||
|
ARROW_PYTHON_STATIC_LIB
|
||||||
|
ARROW_PYTHON_SHARED_LIB
|
||||||
|
ARROW_PYTHON_SHARED_IMP_LIB
|
||||||
|
)
|
||||||
|
else()
|
||||||
|
mark_as_advanced(
|
||||||
|
ARROW_INCLUDE_DIR
|
||||||
|
ARROW_STATIC_LIB
|
||||||
|
ARROW_SHARED_LIB
|
||||||
|
ARROW_PYTHON_STATIC_LIB
|
||||||
|
ARROW_PYTHON_SHARED_LIB
|
||||||
|
)
|
||||||
|
endif()
|
||||||
147
cmake/Modules/FindParquet.cmake
Normal file
147
cmake/Modules/FindParquet.cmake
Normal file
@ -0,0 +1,147 @@
|
|||||||
|
# https://github.com/apache/arrow/blob/master/cpp/cmake_modules/FindParquet.cmake
|
||||||
|
|
||||||
|
# Licensed to the Apache Software Foundation (ASF) under one
|
||||||
|
# or more contributor license agreements. See the NOTICE file
|
||||||
|
# distributed with this work for additional information
|
||||||
|
# regarding copyright ownership. The ASF licenses this file
|
||||||
|
# to you under the Apache License, Version 2.0 (the
|
||||||
|
# "License"); you may not use this file except in compliance
|
||||||
|
# with the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing,
|
||||||
|
# software distributed under the License is distributed on an
|
||||||
|
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||||
|
# KIND, either express or implied. See the License for the
|
||||||
|
# specific language governing permissions and limitations
|
||||||
|
# under the License.
|
||||||
|
|
||||||
|
# - Find PARQUET (parquet/parquet.h, libparquet.a, libparquet.so)
|
||||||
|
# This module defines
|
||||||
|
# PARQUET_INCLUDE_DIR, directory containing headers
|
||||||
|
# PARQUET_LIBS, directory containing parquet libraries
|
||||||
|
# PARQUET_STATIC_LIB, path to libparquet.a
|
||||||
|
# PARQUET_SHARED_LIB, path to libparquet's shared library
|
||||||
|
# PARQUET_SHARED_IMP_LIB, path to libparquet's import library (MSVC only)
|
||||||
|
# PARQUET_FOUND, whether parquet has been found
|
||||||
|
|
||||||
|
include(FindPkgConfig)
|
||||||
|
|
||||||
|
if(NOT "$ENV{PARQUET_HOME}" STREQUAL "")
|
||||||
|
set(PARQUET_HOME "$ENV{PARQUET_HOME}")
|
||||||
|
endif()
|
||||||
|
|
||||||
|
if (MSVC)
|
||||||
|
SET(CMAKE_FIND_LIBRARY_SUFFIXES ".lib" ".dll")
|
||||||
|
|
||||||
|
if (MSVC AND NOT DEFINED PARQUET_MSVC_STATIC_LIB_SUFFIX)
|
||||||
|
set(PARQUET_MSVC_STATIC_LIB_SUFFIX "_static")
|
||||||
|
endif()
|
||||||
|
|
||||||
|
find_library(PARQUET_SHARED_LIBRARIES NAMES parquet
|
||||||
|
PATHS ${PARQUET_HOME} NO_DEFAULT_PATH
|
||||||
|
PATH_SUFFIXES "bin" )
|
||||||
|
|
||||||
|
get_filename_component(PARQUET_SHARED_LIBS ${PARQUET_SHARED_LIBRARIES} PATH )
|
||||||
|
endif ()
|
||||||
|
|
||||||
|
if(PARQUET_HOME)
|
||||||
|
set(PARQUET_SEARCH_HEADER_PATHS
|
||||||
|
${PARQUET_HOME}/include
|
||||||
|
)
|
||||||
|
set(PARQUET_SEARCH_LIB_PATH
|
||||||
|
${PARQUET_HOME}/lib
|
||||||
|
)
|
||||||
|
find_path(PARQUET_INCLUDE_DIR parquet/api/reader.h PATHS
|
||||||
|
${PARQUET_SEARCH_HEADER_PATHS}
|
||||||
|
# make sure we don't accidentally pick up a different version
|
||||||
|
NO_DEFAULT_PATH
|
||||||
|
)
|
||||||
|
find_library(PARQUET_LIBRARIES NAMES parquet
|
||||||
|
PATHS ${PARQUET_HOME} NO_DEFAULT_PATH
|
||||||
|
PATH_SUFFIXES "lib")
|
||||||
|
get_filename_component(PARQUET_LIBS ${PARQUET_LIBRARIES} PATH )
|
||||||
|
|
||||||
|
# Try to autodiscover the Parquet ABI version
|
||||||
|
get_filename_component(PARQUET_LIB_REALPATH ${PARQUET_LIBRARIES} REALPATH)
|
||||||
|
get_filename_component(PARQUET_EXT_REALPATH ${PARQUET_LIB_REALPATH} EXT)
|
||||||
|
string(REGEX MATCH ".([0-9]+.[0-9]+.[0-9]+)" HAS_ABI_VERSION ${PARQUET_EXT_REALPATH})
|
||||||
|
if (HAS_ABI_VERSION)
|
||||||
|
if (APPLE)
|
||||||
|
string(REGEX REPLACE ".([0-9]+.[0-9]+.[0-9]+).dylib" "\\1" PARQUET_ABI_VERSION ${PARQUET_EXT_REALPATH})
|
||||||
|
else()
|
||||||
|
string(REGEX REPLACE ".so.([0-9]+.[0-9]+.[0-9]+)" "\\1" PARQUET_ABI_VERSION ${PARQUET_EXT_REALPATH})
|
||||||
|
endif()
|
||||||
|
string(REGEX REPLACE "([0-9]+).[0-9]+.[0-9]+" "\\1" PARQUET_SO_VERSION ${PARQUET_ABI_VERSION})
|
||||||
|
else()
|
||||||
|
set(PARQUET_ABI_VERSION "1.0.0")
|
||||||
|
set(PARQUET_SO_VERSION "1")
|
||||||
|
endif()
|
||||||
|
else()
|
||||||
|
pkg_check_modules(PARQUET parquet)
|
||||||
|
if (PARQUET_FOUND)
|
||||||
|
pkg_get_variable(PARQUET_ABI_VERSION parquet abi_version)
|
||||||
|
message(STATUS "Parquet C++ ABI version: ${PARQUET_ABI_VERSION}")
|
||||||
|
pkg_get_variable(PARQUET_SO_VERSION parquet so_version)
|
||||||
|
message(STATUS "Parquet C++ SO version: ${PARQUET_SO_VERSION}")
|
||||||
|
set(PARQUET_INCLUDE_DIR ${PARQUET_INCLUDE_DIRS})
|
||||||
|
set(PARQUET_LIBS ${PARQUET_LIBRARY_DIRS})
|
||||||
|
set(PARQUET_SEARCH_LIB_PATH ${PARQUET_LIBRARY_DIRS})
|
||||||
|
message(STATUS "Searching for parquet libs in: ${PARQUET_SEARCH_LIB_PATH}")
|
||||||
|
find_library(PARQUET_LIBRARIES NAMES parquet
|
||||||
|
PATHS ${PARQUET_SEARCH_LIB_PATH} NO_DEFAULT_PATH)
|
||||||
|
else()
|
||||||
|
find_path(PARQUET_INCLUDE_DIR NAMES parquet/api/reader.h )
|
||||||
|
find_library(PARQUET_LIBRARIES NAMES parquet)
|
||||||
|
get_filename_component(PARQUET_LIBS ${PARQUET_LIBRARIES} PATH )
|
||||||
|
endif()
|
||||||
|
endif()
|
||||||
|
|
||||||
|
if (PARQUET_INCLUDE_DIR AND PARQUET_LIBRARIES)
|
||||||
|
set(PARQUET_FOUND TRUE)
|
||||||
|
set(PARQUET_LIB_NAME parquet)
|
||||||
|
if (MSVC)
|
||||||
|
set(PARQUET_STATIC_LIB "${PARQUET_LIBS}/${PARQUET_LIB_NAME}${PARQUET_MSVC_STATIC_LIB_SUFFIX}${CMAKE_STATIC_LIBRARY_SUFFIX}")
|
||||||
|
set(PARQUET_SHARED_LIB "${PARQUET_SHARED_LIBS}/${PARQUET_LIB_NAME}${CMAKE_SHARED_LIBRARY_SUFFIX}")
|
||||||
|
set(PARQUET_SHARED_IMP_LIB "${PARQUET_LIBS}/${PARQUET_LIB_NAME}.lib")
|
||||||
|
else()
|
||||||
|
set(PARQUET_STATIC_LIB ${PARQUET_LIBS}/${CMAKE_STATIC_LIBRARY_PREFIX}${PARQUET_LIB_NAME}.a)
|
||||||
|
set(PARQUET_SHARED_LIB ${PARQUET_LIBS}/${CMAKE_SHARED_LIBRARY_PREFIX}${PARQUET_LIB_NAME}${CMAKE_SHARED_LIBRARY_SUFFIX})
|
||||||
|
endif()
|
||||||
|
else ()
|
||||||
|
set(PARQUET_FOUND FALSE)
|
||||||
|
endif ()
|
||||||
|
|
||||||
|
if (PARQUET_FOUND)
|
||||||
|
if (NOT Parquet_FIND_QUIETLY)
|
||||||
|
message(STATUS "Found the Parquet library: ${PARQUET_LIBRARIES}")
|
||||||
|
endif ()
|
||||||
|
else ()
|
||||||
|
if (NOT Parquet_FIND_QUIETLY)
|
||||||
|
if (NOT PARQUET_FOUND)
|
||||||
|
set(PARQUET_ERR_MSG "${PARQUET_ERR_MSG} Could not find the parquet library.")
|
||||||
|
endif()
|
||||||
|
|
||||||
|
set(PARQUET_ERR_MSG "${PARQUET_ERR_MSG} Looked in ")
|
||||||
|
if ( _parquet_roots )
|
||||||
|
set(PARQUET_ERR_MSG "${PARQUET_ERR_MSG} in ${_parquet_roots}.")
|
||||||
|
else ()
|
||||||
|
set(PARQUET_ERR_MSG "${PARQUET_ERR_MSG} system search paths.")
|
||||||
|
endif ()
|
||||||
|
if (Parquet_FIND_REQUIRED)
|
||||||
|
message(FATAL_ERROR "${PARQUET_ERR_MSG}")
|
||||||
|
else (Parquet_FIND_REQUIRED)
|
||||||
|
message(STATUS "${PARQUET_ERR_MSG}")
|
||||||
|
endif (Parquet_FIND_REQUIRED)
|
||||||
|
endif ()
|
||||||
|
endif ()
|
||||||
|
|
||||||
|
mark_as_advanced(
|
||||||
|
PARQUET_FOUND
|
||||||
|
PARQUET_INCLUDE_DIR
|
||||||
|
PARQUET_LIBS
|
||||||
|
PARQUET_LIBRARIES
|
||||||
|
PARQUET_STATIC_LIB
|
||||||
|
PARQUET_SHARED_LIB
|
||||||
|
)
|
||||||
@ -28,7 +28,7 @@ find_library(METROHASH_LIBRARIES
|
|||||||
|
|
||||||
find_path(METROHASH_INCLUDE_DIR
|
find_path(METROHASH_INCLUDE_DIR
|
||||||
NAMES metrohash.h
|
NAMES metrohash.h
|
||||||
PATHS ${METROHASH_ROOT_DIR}/include ${METROHASH_INCLUDE_PATHS}
|
PATHS ${METROHASH_ROOT_DIR}/include PATH_SUFFIXES metrohash ${METROHASH_INCLUDE_PATHS}
|
||||||
)
|
)
|
||||||
|
|
||||||
include(FindPackageHandleStandardArgs)
|
include(FindPackageHandleStandardArgs)
|
||||||
|
|||||||
@ -24,3 +24,10 @@ if (CMAKE_CXX_COMPILER_ID STREQUAL "GNU")
|
|||||||
elseif (CMAKE_CXX_COMPILER_ID STREQUAL "Clang")
|
elseif (CMAKE_CXX_COMPILER_ID STREQUAL "Clang")
|
||||||
set (COMPILER_CLANG 1)
|
set (COMPILER_CLANG 1)
|
||||||
endif ()
|
endif ()
|
||||||
|
|
||||||
|
if (CMAKE_SYSTEM_PROCESSOR MATCHES "^(ppc64le.*|PPC64LE.*)")
|
||||||
|
set (ARCH_PPC64LE 1)
|
||||||
|
if (COMPILER_CLANG OR (COMPILER_GCC AND CMAKE_CXX_COMPILER_VERSION VERSION_LESS 8))
|
||||||
|
message(FATAL_ERROR "Only gcc-8 is supported for powerpc architecture")
|
||||||
|
endif ()
|
||||||
|
endif ()
|
||||||
|
|||||||
@ -9,7 +9,7 @@ endif ()
|
|||||||
if (NOT USE_INTERNAL_BOOST_LIBRARY)
|
if (NOT USE_INTERNAL_BOOST_LIBRARY)
|
||||||
set (Boost_USE_STATIC_LIBS ${USE_STATIC_LIBRARIES})
|
set (Boost_USE_STATIC_LIBS ${USE_STATIC_LIBRARIES})
|
||||||
set (BOOST_ROOT "/usr/local")
|
set (BOOST_ROOT "/usr/local")
|
||||||
find_package (Boost 1.60 COMPONENTS program_options system filesystem thread)
|
find_package (Boost 1.60 COMPONENTS program_options system filesystem thread regex)
|
||||||
# incomplete, no include search, who use it?
|
# incomplete, no include search, who use it?
|
||||||
if (NOT Boost_FOUND)
|
if (NOT Boost_FOUND)
|
||||||
# # Try to find manually.
|
# # Try to find manually.
|
||||||
@ -29,9 +29,12 @@ if (NOT Boost_SYSTEM_LIBRARY)
|
|||||||
set (Boost_SYSTEM_LIBRARY boost_system_internal)
|
set (Boost_SYSTEM_LIBRARY boost_system_internal)
|
||||||
set (Boost_PROGRAM_OPTIONS_LIBRARY boost_program_options_internal)
|
set (Boost_PROGRAM_OPTIONS_LIBRARY boost_program_options_internal)
|
||||||
set (Boost_FILESYSTEM_LIBRARY boost_filesystem_internal ${Boost_SYSTEM_LIBRARY})
|
set (Boost_FILESYSTEM_LIBRARY boost_filesystem_internal ${Boost_SYSTEM_LIBRARY})
|
||||||
|
set (Boost_REGEX_LIBRARY boost_regex_internal)
|
||||||
|
|
||||||
set (Boost_INCLUDE_DIRS)
|
set (Boost_INCLUDE_DIRS)
|
||||||
|
|
||||||
|
set (BOOST_ROOT "${ClickHouse_SOURCE_DIR}/contrib/boost")
|
||||||
|
|
||||||
# For boost from github:
|
# For boost from github:
|
||||||
file (GLOB Boost_INCLUDE_DIRS_ "${ClickHouse_SOURCE_DIR}/contrib/boost/libs/*/include")
|
file (GLOB Boost_INCLUDE_DIRS_ "${ClickHouse_SOURCE_DIR}/contrib/boost/libs/*/include")
|
||||||
list (APPEND Boost_INCLUDE_DIRS ${Boost_INCLUDE_DIRS_})
|
list (APPEND Boost_INCLUDE_DIRS ${Boost_INCLUDE_DIRS_})
|
||||||
@ -44,4 +47,4 @@ if (NOT Boost_SYSTEM_LIBRARY)
|
|||||||
|
|
||||||
endif ()
|
endif ()
|
||||||
|
|
||||||
message (STATUS "Using Boost: ${Boost_INCLUDE_DIRS} : ${Boost_PROGRAM_OPTIONS_LIBRARY},${Boost_SYSTEM_LIBRARY},${Boost_FILESYSTEM_LIBRARY}")
|
message (STATUS "Using Boost: ${Boost_INCLUDE_DIRS} : ${Boost_PROGRAM_OPTIONS_LIBRARY},${Boost_SYSTEM_LIBRARY},${Boost_FILESYSTEM_LIBRARY},${Boost_REGEX_LIBRARY}")
|
||||||
|
|||||||
30
cmake/find_brotli.cmake
Normal file
30
cmake/find_brotli.cmake
Normal file
@ -0,0 +1,30 @@
|
|||||||
|
option (USE_INTERNAL_BROTLI_LIBRARY "Set to FALSE to use system libbrotli library instead of bundled" ${NOT_UNBUNDLED})
|
||||||
|
|
||||||
|
if (NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/brotli/c/include/brotli/decode.h")
|
||||||
|
if (USE_INTERNAL_BROTLI_LIBRARY)
|
||||||
|
message (WARNING "submodule contrib/brotli is missing. to fix try run: \n git submodule update --init --recursive")
|
||||||
|
set (USE_INTERNAL_BROTLI_LIBRARY 0)
|
||||||
|
endif ()
|
||||||
|
set (MISSING_INTERNAL_BROTLI_LIBRARY 1)
|
||||||
|
endif ()
|
||||||
|
|
||||||
|
if(NOT USE_INTERNAL_BROTLI_LIBRARY)
|
||||||
|
find_library(BROTLI_LIBRARY_COMMON brotlicommon)
|
||||||
|
find_library(BROTLI_LIBRARY_DEC brotlidec)
|
||||||
|
find_library(BROTLI_LIBRARY_ENC brotlienc)
|
||||||
|
find_path(BROTLI_INCLUDE_DIR NAMES brotli/decode.h brotli/encode.h brotli/port.h brotli/types.h PATHS ${BROTLI_INCLUDE_PATHS})
|
||||||
|
if(BROTLI_LIBRARY_DEC AND BROTLI_LIBRARY_ENC AND BROTLI_LIBRARY_COMMON)
|
||||||
|
set(BROTLI_LIBRARY ${BROTLI_LIBRARY_DEC} ${BROTLI_LIBRARY_ENC} ${BROTLI_LIBRARY_COMMON})
|
||||||
|
endif()
|
||||||
|
endif()
|
||||||
|
|
||||||
|
if (BROTLI_LIBRARY AND BROTLI_INCLUDE_DIR)
|
||||||
|
set (USE_BROTLI 1)
|
||||||
|
elseif (NOT MISSING_INTERNAL_BROTLI_LIBRARY)
|
||||||
|
set (BROTLI_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/contrib/brotli/c/include)
|
||||||
|
set (USE_INTERNAL_BROTLI_LIBRARY 1)
|
||||||
|
set (BROTLI_LIBRARY brotli)
|
||||||
|
set (USE_BROTLI 1)
|
||||||
|
endif ()
|
||||||
|
|
||||||
|
message (STATUS "Using brotli=${USE_BROTLI}: ${BROTLI_INCLUDE_DIR} : ${BROTLI_LIBRARY}")
|
||||||
@ -1,12 +1,13 @@
|
|||||||
|
# ARM: Cannot cpuid_get_raw_data: CPUID instruction is not supported
|
||||||
if (NOT ARCH_ARM)
|
if (NOT ARCH_ARM)
|
||||||
option (USE_INTERNAL_CPUID_LIBRARY "Set to FALSE to use system cpuid library instead of bundled" ${NOT_UNBUNDLED})
|
option (USE_INTERNAL_CPUID_LIBRARY "Set to FALSE to use system cpuid library instead of bundled" ${NOT_UNBUNDLED})
|
||||||
endif ()
|
endif ()
|
||||||
|
|
||||||
#if (USE_INTERNAL_CPUID_LIBRARY AND NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/libcpuid/include/cpuid/libcpuid.h")
|
if (USE_INTERNAL_CPUID_LIBRARY AND NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/libcpuid/CMakeLists.txt")
|
||||||
# message (WARNING "submodule contrib/libcpuid is missing. to fix try run: \n git submodule update --init --recursive")
|
message (WARNING "submodule contrib/libcpuid is missing. to fix try run: \n git submodule update --init --recursive")
|
||||||
# set (USE_INTERNAL_CPUID_LIBRARY 0)
|
set (USE_INTERNAL_CPUID_LIBRARY 0)
|
||||||
# set (MISSING_INTERNAL_CPUID_LIBRARY 1)
|
set (MISSING_INTERNAL_CPUID_LIBRARY 1)
|
||||||
#endif ()
|
endif ()
|
||||||
|
|
||||||
if (NOT USE_INTERNAL_CPUID_LIBRARY)
|
if (NOT USE_INTERNAL_CPUID_LIBRARY)
|
||||||
find_library (CPUID_LIBRARY cpuid)
|
find_library (CPUID_LIBRARY cpuid)
|
||||||
@ -20,10 +21,12 @@ if (CPUID_LIBRARY AND CPUID_INCLUDE_DIR)
|
|||||||
add_definitions(-DHAVE_STDINT_H)
|
add_definitions(-DHAVE_STDINT_H)
|
||||||
# TODO: make virtual target cpuid:cpuid with COMPILE_DEFINITIONS property
|
# TODO: make virtual target cpuid:cpuid with COMPILE_DEFINITIONS property
|
||||||
endif ()
|
endif ()
|
||||||
elseif (NOT MISSING_INTERNAL_CPUID_LIBRARY)
|
set (USE_CPUID 1)
|
||||||
|
elseif (NOT ARCH_ARM AND NOT MISSING_INTERNAL_CPUID_LIBRARY)
|
||||||
set (CPUID_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/contrib/libcpuid/include)
|
set (CPUID_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/contrib/libcpuid/include)
|
||||||
set (USE_INTERNAL_CPUID_LIBRARY 1)
|
set (USE_INTERNAL_CPUID_LIBRARY 1)
|
||||||
set (CPUID_LIBRARY cpuid)
|
set (CPUID_LIBRARY cpuid)
|
||||||
|
set (USE_CPUID 1)
|
||||||
endif ()
|
endif ()
|
||||||
|
|
||||||
message (STATUS "Using cpuid: ${CPUID_INCLUDE_DIR} : ${CPUID_LIBRARY}")
|
message (STATUS "Using cpuid=${USE_CPUID}: ${CPUID_INCLUDE_DIR} : ${CPUID_LIBRARY}")
|
||||||
|
|||||||
24
cmake/find_cpuinfo.cmake
Normal file
24
cmake/find_cpuinfo.cmake
Normal file
@ -0,0 +1,24 @@
|
|||||||
|
option(USE_INTERNAL_CPUINFO_LIBRARY "Set to FALSE to use system cpuinfo library instead of bundled" ${NOT_UNBUNDLED})
|
||||||
|
|
||||||
|
# Now we have no contrib/libcpuinfo, use from system.
|
||||||
|
if (USE_INTERNAL_CPUINFO_LIBRARY AND NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/libcpuinfo/include")
|
||||||
|
#message (WARNING "submodule contrib/libcpuid is missing. to fix try run: \n git submodule update --init --recursive")
|
||||||
|
set (USE_INTERNAL_CPUINFO_LIBRARY 0)
|
||||||
|
set (MISSING_INTERNAL_CPUINFO_LIBRARY 1)
|
||||||
|
endif ()
|
||||||
|
|
||||||
|
if(NOT USE_INTERNAL_CPUINFO_LIBRARY)
|
||||||
|
find_library(CPUINFO_LIBRARY cpuinfo)
|
||||||
|
find_path(CPUINFO_INCLUDE_DIR NAMES cpuinfo.h PATHS ${CPUINFO_INCLUDE_PATHS})
|
||||||
|
endif()
|
||||||
|
|
||||||
|
if(CPUID_LIBRARY AND CPUID_INCLUDE_DIR)
|
||||||
|
set(USE_CPUINFO 1)
|
||||||
|
elseif(NOT MISSING_INTERNAL_CPUINFO_LIBRARY)
|
||||||
|
set(CPUINFO_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/contrib/libcpuinfo/include)
|
||||||
|
set(USE_INTERNAL_CPUINFO_LIBRARY 1)
|
||||||
|
set(CPUINFO_LIBRARY cpuinfo)
|
||||||
|
set(USE_CPUINFO 1)
|
||||||
|
endif()
|
||||||
|
|
||||||
|
message(STATUS "Using cpuinfo=${USE_CPUINFO}: ${CPUINFO_INCLUDE_DIR} : ${CPUINFO_LIBRARY}")
|
||||||
@ -8,13 +8,23 @@ if (NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/googletest/googletest/CMakeList
|
|||||||
set (MISSING_INTERNAL_GTEST_LIBRARY 1)
|
set (MISSING_INTERNAL_GTEST_LIBRARY 1)
|
||||||
endif ()
|
endif ()
|
||||||
|
|
||||||
|
|
||||||
if(NOT USE_INTERNAL_GTEST_LIBRARY)
|
if(NOT USE_INTERNAL_GTEST_LIBRARY)
|
||||||
|
# TODO: autodetect of GTEST_SRC_DIR by EXISTS /usr/src/googletest/CMakeLists.txt
|
||||||
|
if(NOT GTEST_SRC_DIR)
|
||||||
find_package(GTest)
|
find_package(GTest)
|
||||||
endif()
|
endif()
|
||||||
|
|
||||||
if (NOT GTEST_INCLUDE_DIRS AND NOT MISSING_INTERNAL_GTEST_LIBRARY)
|
|
||||||
set (USE_INTERNAL_GTEST_LIBRARY 1)
|
|
||||||
set (GTEST_MAIN_LIBRARIES gtest_main)
|
|
||||||
endif()
|
endif()
|
||||||
|
|
||||||
message (STATUS "Using gtest: ${GTEST_INCLUDE_DIRS} : ${GTEST_MAIN_LIBRARIES}")
|
if (NOT GTEST_SRC_DIR AND NOT GTEST_INCLUDE_DIRS AND NOT MISSING_INTERNAL_GTEST_LIBRARY)
|
||||||
|
set (USE_INTERNAL_GTEST_LIBRARY 1)
|
||||||
|
set (GTEST_MAIN_LIBRARIES gtest_main)
|
||||||
|
set (GTEST_LIBRARIES gtest)
|
||||||
|
set (GTEST_INCLUDE_DIRS ${ClickHouse_SOURCE_DIR}/contrib/googletest/googletest)
|
||||||
|
endif ()
|
||||||
|
|
||||||
|
if((GTEST_INCLUDE_DIRS AND GTEST_MAIN_LIBRARIES) OR GTEST_SRC_DIR)
|
||||||
|
set(USE_GTEST 1)
|
||||||
|
endif()
|
||||||
|
|
||||||
|
message (STATUS "Using gtest=${USE_GTEST}: ${GTEST_INCLUDE_DIRS} : ${GTEST_LIBRARIES}, ${GTEST_MAIN_LIBRARIES} : ${GTEST_SRC_DIR}")
|
||||||
|
|||||||
@ -1,4 +1,4 @@
|
|||||||
if (NOT ARCH_ARM AND NOT OS_FREEBSD AND NOT APPLE)
|
if (NOT ARCH_ARM AND NOT OS_FREEBSD AND NOT APPLE AND USE_PROTOBUF)
|
||||||
option (ENABLE_HDFS "Enable HDFS" ${NOT_UNBUNDLED})
|
option (ENABLE_HDFS "Enable HDFS" ${NOT_UNBUNDLED})
|
||||||
endif ()
|
endif ()
|
||||||
|
|
||||||
|
|||||||
@ -9,8 +9,9 @@ if (NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/libxml2/libxml.h")
|
|||||||
endif ()
|
endif ()
|
||||||
|
|
||||||
if (NOT USE_INTERNAL_LIBXML2_LIBRARY)
|
if (NOT USE_INTERNAL_LIBXML2_LIBRARY)
|
||||||
find_library (LIBXML2_LIBRARY libxml2)
|
find_package (LibXml2)
|
||||||
find_path (LIBXML2_INCLUDE_DIR NAMES libxml.h PATHS ${LIBXML2_INCLUDE_PATHS})
|
#find_library (LIBXML2_LIBRARY libxml2)
|
||||||
|
#find_path (LIBXML2_INCLUDE_DIR NAMES libxml.h PATHS ${LIBXML2_INCLUDE_PATHS})
|
||||||
endif ()
|
endif ()
|
||||||
|
|
||||||
if (LIBXML2_LIBRARY AND LIBXML2_INCLUDE_DIR)
|
if (LIBXML2_LIBRARY AND LIBXML2_INCLUDE_DIR)
|
||||||
|
|||||||
@ -22,6 +22,9 @@ if (ENABLE_EMBEDDED_COMPILER)
|
|||||||
if (NOT LLVM_FOUND)
|
if (NOT LLVM_FOUND)
|
||||||
find_package (LLVM 6 CONFIG PATHS ${LLVM_PATHS})
|
find_package (LLVM 6 CONFIG PATHS ${LLVM_PATHS})
|
||||||
endif ()
|
endif ()
|
||||||
|
if (NOT LLVM_FOUND)
|
||||||
|
find_package (LLVM 8 CONFIG PATHS ${LLVM_PATHS})
|
||||||
|
endif ()
|
||||||
if (NOT LLVM_FOUND)
|
if (NOT LLVM_FOUND)
|
||||||
find_package (LLVM 5 CONFIG PATHS ${LLVM_PATHS})
|
find_package (LLVM 5 CONFIG PATHS ${LLVM_PATHS})
|
||||||
endif ()
|
endif ()
|
||||||
@ -79,7 +82,11 @@ endif()
|
|||||||
|
|
||||||
function(llvm_libs_all REQUIRED_LLVM_LIBRARIES)
|
function(llvm_libs_all REQUIRED_LLVM_LIBRARIES)
|
||||||
llvm_map_components_to_libnames (result all)
|
llvm_map_components_to_libnames (result all)
|
||||||
|
if (USE_STATIC_LIBRARIES OR NOT "LLVM" IN_LIST result)
|
||||||
list (REMOVE_ITEM result "LTO" "LLVM")
|
list (REMOVE_ITEM result "LTO" "LLVM")
|
||||||
|
else()
|
||||||
|
set (result "LLVM")
|
||||||
|
endif ()
|
||||||
if (TERMCAP_LIBRARY)
|
if (TERMCAP_LIBRARY)
|
||||||
list (APPEND result ${TERMCAP_LIBRARY})
|
list (APPEND result ${TERMCAP_LIBRARY})
|
||||||
endif ()
|
endif ()
|
||||||
|
|||||||
@ -1,9 +1,12 @@
|
|||||||
option (USE_INTERNAL_LZ4_LIBRARY "Set to FALSE to use system lz4 library instead of bundled" ${NOT_UNBUNDLED})
|
option (USE_INTERNAL_LZ4_LIBRARY "Set to FALSE to use system lz4 library instead of bundled" ${NOT_UNBUNDLED})
|
||||||
|
|
||||||
if (USE_INTERNAL_LZ4_LIBRARY AND NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/lz4/lib/lz4.h")
|
if (NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/lz4/lib/lz4.h")
|
||||||
|
if (USE_INTERNAL_LZ4_LIBRARY)
|
||||||
message (WARNING "submodule contrib/lz4 is missing. to fix try run: \n git submodule update --init --recursive")
|
message (WARNING "submodule contrib/lz4 is missing. to fix try run: \n git submodule update --init --recursive")
|
||||||
set (USE_INTERNAL_LZ4_LIBRARY 0)
|
set (USE_INTERNAL_LZ4_LIBRARY 0)
|
||||||
endif ()
|
endif ()
|
||||||
|
set (MISSING_INTERNAL_LZ4_LIBRARY 1)
|
||||||
|
endif ()
|
||||||
|
|
||||||
if (NOT USE_INTERNAL_LZ4_LIBRARY)
|
if (NOT USE_INTERNAL_LZ4_LIBRARY)
|
||||||
find_library (LZ4_LIBRARY lz4)
|
find_library (LZ4_LIBRARY lz4)
|
||||||
@ -11,7 +14,7 @@ if (NOT USE_INTERNAL_LZ4_LIBRARY)
|
|||||||
endif ()
|
endif ()
|
||||||
|
|
||||||
if (LZ4_LIBRARY AND LZ4_INCLUDE_DIR)
|
if (LZ4_LIBRARY AND LZ4_INCLUDE_DIR)
|
||||||
else ()
|
elseif (NOT MISSING_INTERNAL_LZ4_LIBRARY)
|
||||||
set (LZ4_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/contrib/lz4/lib)
|
set (LZ4_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/contrib/lz4/lib)
|
||||||
set (USE_INTERNAL_LZ4_LIBRARY 1)
|
set (USE_INTERNAL_LZ4_LIBRARY 1)
|
||||||
set (LZ4_LIBRARY lz4)
|
set (LZ4_LIBRARY lz4)
|
||||||
|
|||||||
68
cmake/find_parquet.cmake
Normal file
68
cmake/find_parquet.cmake
Normal file
@ -0,0 +1,68 @@
|
|||||||
|
if (NOT OS_FREEBSD) # Freebsd: ../contrib/arrow/cpp/src/arrow/util/bit-util.h:27:10: fatal error: endian.h: No such file or directory
|
||||||
|
option(USE_INTERNAL_PARQUET_LIBRARY "Set to FALSE to use system parquet library instead of bundled" ${NOT_UNBUNDLED})
|
||||||
|
endif()
|
||||||
|
|
||||||
|
if(NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/arrow/cpp/CMakeLists.txt")
|
||||||
|
if(USE_INTERNAL_PARQUET_LIBRARY)
|
||||||
|
message(WARNING "submodule contrib/arrow (required for Parquet) is missing. to fix try run: \n git submodule update --init --recursive")
|
||||||
|
endif()
|
||||||
|
set(USE_INTERNAL_PARQUET_LIBRARY 0)
|
||||||
|
set(MISSING_INTERNAL_PARQUET_LIBRARY 1)
|
||||||
|
endif()
|
||||||
|
|
||||||
|
if(NOT USE_INTERNAL_PARQUET_LIBRARY)
|
||||||
|
find_package(Arrow)
|
||||||
|
find_package(Parquet)
|
||||||
|
endif()
|
||||||
|
|
||||||
|
if(ARROW_INCLUDE_DIR AND PARQUET_INCLUDE_DIR)
|
||||||
|
elseif(NOT MISSING_INTERNAL_PARQUET_LIBRARY AND NOT OS_FREEBSD)
|
||||||
|
include(cmake/find_snappy.cmake)
|
||||||
|
set(CAN_USE_INTERNAL_PARQUET_LIBRARY 1)
|
||||||
|
include(CheckCXXSourceCompiles)
|
||||||
|
if(NOT USE_INTERNAL_DOUBLE_CONVERSION_LIBRARY)
|
||||||
|
set(CMAKE_REQUIRED_LIBRARIES ${DOUBLE_CONVERSION_LIBRARIES})
|
||||||
|
set(CMAKE_REQUIRED_INCLUDES ${DOUBLE_CONVERSION_INCLUDE_DIR})
|
||||||
|
check_cxx_source_compiles("
|
||||||
|
#include <double-conversion/double-conversion.h>
|
||||||
|
int main() { static const int flags_ = double_conversion::StringToDoubleConverter::ALLOW_CASE_INSENSIBILITY; return 0;}
|
||||||
|
" HAVE_DOUBLE_CONVERSION_ALLOW_CASE_INSENSIBILITY)
|
||||||
|
|
||||||
|
if(NOT HAVE_DOUBLE_CONVERSION_ALLOW_CASE_INSENSIBILITY) # HAVE_STD_RANDOM_SHUFFLE
|
||||||
|
message(STATUS "Disabling internal parquet library because arrow is broken (can't use old double_conversion)")
|
||||||
|
set(CAN_USE_INTERNAL_PARQUET_LIBRARY 0)
|
||||||
|
endif()
|
||||||
|
endif()
|
||||||
|
|
||||||
|
if(NOT CAN_USE_INTERNAL_PARQUET_LIBRARY)
|
||||||
|
set(USE_INTERNAL_PARQUET_LIBRARY 0)
|
||||||
|
else()
|
||||||
|
set(USE_INTERNAL_PARQUET_LIBRARY 1)
|
||||||
|
|
||||||
|
if(USE_INTERNAL_PARQUET_LIBRARY_NATIVE_CMAKE)
|
||||||
|
set(ARROW_INCLUDE_DIR "${ClickHouse_SOURCE_DIR}/contrib/arrow/cpp/src")
|
||||||
|
set(PARQUET_INCLUDE_DIR "${ClickHouse_SOURCE_DIR}/contrib/arrow/cpp/src" ${ClickHouse_BINARY_DIR}/contrib/arrow/cpp/src)
|
||||||
|
endif()
|
||||||
|
|
||||||
|
if(${USE_STATIC_LIBRARIES})
|
||||||
|
set(ARROW_LIBRARY arrow_static)
|
||||||
|
set(PARQUET_LIBRARY parquet_static)
|
||||||
|
set(THRIFT_LIBRARY thrift_static)
|
||||||
|
else()
|
||||||
|
set(ARROW_LIBRARY arrow_shared)
|
||||||
|
set(PARQUET_LIBRARY parquet_shared)
|
||||||
|
if(USE_INTERNAL_PARQUET_LIBRARY_NATIVE_CMAKE)
|
||||||
|
list(APPEND PARQUET_LIBRARY ${Boost_REGEX_LIBRARY})
|
||||||
|
endif()
|
||||||
|
set(THRIFT_LIBRARY thrift)
|
||||||
|
endif()
|
||||||
|
|
||||||
|
set(USE_PARQUET 1)
|
||||||
|
endif()
|
||||||
|
endif()
|
||||||
|
|
||||||
|
if(USE_PARQUET)
|
||||||
|
message(STATUS "Using Parquet: ${ARROW_LIBRARY}:${ARROW_INCLUDE_DIR} ; ${PARQUET_LIBRARY}:${PARQUET_INCLUDE_DIR} ; ${THRIFT_LIBRARY}")
|
||||||
|
else()
|
||||||
|
message(STATUS "Building without Parquet support")
|
||||||
|
endif()
|
||||||
2
cmake/find_pdqsort.cmake
Normal file
2
cmake/find_pdqsort.cmake
Normal file
@ -0,0 +1,2 @@
|
|||||||
|
set(PDQSORT_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/contrib/pdqsort)
|
||||||
|
message(STATUS "Using pdqsort: ${PDQSORT_INCLUDE_DIR}")
|
||||||
@ -1,18 +1,35 @@
|
|||||||
option (USE_INTERNAL_PROTOBUF_LIBRARY "Set to FALSE to use system protobuf instead of bundled" ON)
|
option(USE_INTERNAL_PROTOBUF_LIBRARY "Set to FALSE to use system protobuf instead of bundled" ${NOT_UNBUNDLED})
|
||||||
|
|
||||||
|
if(OS_FREEBSD AND SANITIZE STREQUAL "address")
|
||||||
|
# ../contrib/protobuf/src/google/protobuf/arena_impl.h:45:10: fatal error: 'sanitizer/asan_interface.h' file not found
|
||||||
|
set(MISSING_INTERNAL_PROTOBUF_LIBRARY 1)
|
||||||
|
set(USE_INTERNAL_PROTOBUF_LIBRARY 0)
|
||||||
|
endif()
|
||||||
|
|
||||||
|
if(NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/protobuf/cmake/CMakeLists.txt")
|
||||||
|
if(USE_INTERNAL_PROTOBUF_LIBRARY)
|
||||||
|
message(WARNING "submodule contrib/protobuf is missing. to fix try run: \n git submodule update --init --recursive")
|
||||||
|
set(USE_INTERNAL_PROTOBUF_LIBRARY 0)
|
||||||
|
endif()
|
||||||
|
set(MISSING_INTERNAL_PROTOBUF_LIBRARY 1)
|
||||||
|
endif()
|
||||||
|
|
||||||
if(NOT USE_INTERNAL_PROTOBUF_LIBRARY)
|
if(NOT USE_INTERNAL_PROTOBUF_LIBRARY)
|
||||||
find_package(Protobuf)
|
find_package(Protobuf)
|
||||||
endif()
|
endif()
|
||||||
|
|
||||||
if (Protobuf_LIBRARY AND Protobuf_INCLUDE_DIR)
|
if (Protobuf_LIBRARY AND Protobuf_INCLUDE_DIR)
|
||||||
else ()
|
set(USE_PROTOBUF 1)
|
||||||
set(Protobuf_INCLUDE_DIR ${CMAKE_SOURCE_DIR}/contrib/protobuf/src)
|
elseif(NOT MISSING_INTERNAL_PROTOBUF_LIBRARY)
|
||||||
|
set(Protobuf_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/contrib/protobuf/src)
|
||||||
|
|
||||||
|
set(USE_PROTOBUF 1)
|
||||||
|
set(USE_INTERNAL_PROTOBUF_LIBRARY 1)
|
||||||
set(Protobuf_LIBRARY libprotobuf)
|
set(Protobuf_LIBRARY libprotobuf)
|
||||||
set(Protobuf_PROTOC_LIBRARY libprotoc)
|
set(Protobuf_PROTOC_LIBRARY libprotoc)
|
||||||
set(Protobuf_LITE_LIBRARY libprotobuf-lite)
|
set(Protobuf_LITE_LIBRARY libprotobuf-lite)
|
||||||
|
|
||||||
set(Protobuf_PROTOC_EXECUTABLE ${CMAKE_BINARY_DIR}/contrib/protobuf/cmake/protoc)
|
set(Protobuf_PROTOC_EXECUTABLE ${ClickHouse_BINARY_DIR}/contrib/protobuf/cmake/protoc)
|
||||||
|
|
||||||
if(NOT DEFINED PROTOBUF_GENERATE_CPP_APPEND_PATH)
|
if(NOT DEFINED PROTOBUF_GENERATE_CPP_APPEND_PATH)
|
||||||
set(PROTOBUF_GENERATE_CPP_APPEND_PATH TRUE)
|
set(PROTOBUF_GENERATE_CPP_APPEND_PATH TRUE)
|
||||||
@ -77,4 +94,4 @@ else ()
|
|||||||
endfunction()
|
endfunction()
|
||||||
endif()
|
endif()
|
||||||
|
|
||||||
message (STATUS "Using protobuf: ${Protobuf_INCLUDE_DIR} : ${Protobuf_LIBRARY}")
|
message(STATUS "Using protobuf=${USE_PROTOBUF}: ${Protobuf_INCLUDE_DIR} : ${Protobuf_LIBRARY}")
|
||||||
|
|||||||
@ -1,7 +1,13 @@
|
|||||||
if (NOT ARCH_ARM AND NOT ARCH_32)
|
# Freebsd: contrib/cppkafka/include/cppkafka/detail/endianness.h:53:23: error: 'betoh16' was not declared in this scope
|
||||||
|
if (NOT ARCH_ARM AND NOT ARCH_32 AND NOT APPLE AND NOT OS_FREEBSD)
|
||||||
option (ENABLE_RDKAFKA "Enable kafka" ON)
|
option (ENABLE_RDKAFKA "Enable kafka" ON)
|
||||||
endif ()
|
endif ()
|
||||||
|
|
||||||
|
if (NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/cppkafka/CMakeLists.txt")
|
||||||
|
message (WARNING "submodule contrib/cppkafka is missing. to fix try run: \n git submodule update --init --recursive")
|
||||||
|
set (ENABLE_RDKAFKA 0)
|
||||||
|
endif ()
|
||||||
|
|
||||||
if (ENABLE_RDKAFKA)
|
if (ENABLE_RDKAFKA)
|
||||||
|
|
||||||
if (OS_LINUX AND NOT ARCH_ARM)
|
if (OS_LINUX AND NOT ARCH_ARM)
|
||||||
@ -20,11 +26,13 @@ if (NOT USE_INTERNAL_RDKAFKA_LIBRARY)
|
|||||||
if (USE_STATIC_LIBRARIES AND NOT OS_FREEBSD)
|
if (USE_STATIC_LIBRARIES AND NOT OS_FREEBSD)
|
||||||
find_library (SASL2_LIBRARY sasl2)
|
find_library (SASL2_LIBRARY sasl2)
|
||||||
endif ()
|
endif ()
|
||||||
|
set (CPPKAFKA_LIBRARY cppkafka) # TODO: try to use unbundled version.
|
||||||
endif ()
|
endif ()
|
||||||
|
|
||||||
if (RDKAFKA_LIB AND RDKAFKA_INCLUDE_DIR)
|
if (RDKAFKA_LIB AND RDKAFKA_INCLUDE_DIR)
|
||||||
set (USE_RDKAFKA 1)
|
set (USE_RDKAFKA 1)
|
||||||
set (RDKAFKA_LIBRARY ${RDKAFKA_LIB} ${OPENSSL_LIBRARIES})
|
set (RDKAFKA_LIBRARY ${RDKAFKA_LIB} ${OPENSSL_LIBRARIES})
|
||||||
|
set (CPPKAFKA_LIBRARY cppkafka)
|
||||||
if (SASL2_LIBRARY)
|
if (SASL2_LIBRARY)
|
||||||
list (APPEND RDKAFKA_LIBRARY ${SASL2_LIBRARY})
|
list (APPEND RDKAFKA_LIBRARY ${SASL2_LIBRARY})
|
||||||
endif ()
|
endif ()
|
||||||
@ -35,9 +43,10 @@ elseif (NOT MISSING_INTERNAL_RDKAFKA_LIBRARY AND NOT ARCH_ARM)
|
|||||||
set (USE_INTERNAL_RDKAFKA_LIBRARY 1)
|
set (USE_INTERNAL_RDKAFKA_LIBRARY 1)
|
||||||
set (RDKAFKA_INCLUDE_DIR "${ClickHouse_SOURCE_DIR}/contrib/librdkafka/src")
|
set (RDKAFKA_INCLUDE_DIR "${ClickHouse_SOURCE_DIR}/contrib/librdkafka/src")
|
||||||
set (RDKAFKA_LIBRARY rdkafka)
|
set (RDKAFKA_LIBRARY rdkafka)
|
||||||
|
set (CPPKAFKA_LIBRARY cppkafka)
|
||||||
set (USE_RDKAFKA 1)
|
set (USE_RDKAFKA 1)
|
||||||
endif ()
|
endif ()
|
||||||
|
|
||||||
endif ()
|
endif ()
|
||||||
|
|
||||||
message (STATUS "Using librdkafka=${USE_RDKAFKA}: ${RDKAFKA_INCLUDE_DIR} : ${RDKAFKA_LIBRARY}")
|
message (STATUS "Using librdkafka=${USE_RDKAFKA}: ${RDKAFKA_INCLUDE_DIR} : ${RDKAFKA_LIBRARY} ${CPPKAFKA_LIBRARY}")
|
||||||
|
|||||||
@ -5,13 +5,24 @@ if (NOT USE_INTERNAL_RE2_LIBRARY)
|
|||||||
find_path (RE2_INCLUDE_DIR NAMES re2/re2.h PATHS ${RE2_INCLUDE_PATHS})
|
find_path (RE2_INCLUDE_DIR NAMES re2/re2.h PATHS ${RE2_INCLUDE_PATHS})
|
||||||
endif ()
|
endif ()
|
||||||
|
|
||||||
|
string(FIND ${CMAKE_CURRENT_BINARY_DIR} " " _have_space)
|
||||||
|
if(_have_space GREATER 0)
|
||||||
|
message(WARNING "Using spaces in build path [${CMAKE_CURRENT_BINARY_DIR}] highly not recommended. Library re2st will be disabled.")
|
||||||
|
set (MISSING_INTERNAL_RE2_ST_LIBRARY 1)
|
||||||
|
endif()
|
||||||
|
|
||||||
if (RE2_LIBRARY AND RE2_INCLUDE_DIR)
|
if (RE2_LIBRARY AND RE2_INCLUDE_DIR)
|
||||||
set (RE2_ST_LIBRARY ${RE2_LIBRARY})
|
set (RE2_ST_LIBRARY ${RE2_LIBRARY})
|
||||||
else ()
|
elseif (NOT MISSING_INTERNAL_RE2_LIBRARY)
|
||||||
set (USE_INTERNAL_RE2_LIBRARY 1)
|
set (USE_INTERNAL_RE2_LIBRARY 1)
|
||||||
set (RE2_LIBRARY re2)
|
set (RE2_LIBRARY re2)
|
||||||
|
set (RE2_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/contrib/re2)
|
||||||
|
if (NOT MISSING_INTERNAL_RE2_ST_LIBRARY)
|
||||||
set (RE2_ST_LIBRARY re2_st)
|
set (RE2_ST_LIBRARY re2_st)
|
||||||
set (USE_RE2_ST 1)
|
set (USE_RE2_ST 1)
|
||||||
|
else ()
|
||||||
|
set (RE2_ST_LIBRARY ${RE2_LIBRARY})
|
||||||
|
endif ()
|
||||||
endif ()
|
endif ()
|
||||||
|
|
||||||
message (STATUS "Using re2: ${RE2_INCLUDE_DIR} : ${RE2_LIBRARY}; ${RE2_ST_INCLUDE_DIR} : ${RE2_ST_LIBRARY}")
|
message (STATUS "Using re2: ${RE2_INCLUDE_DIR} : ${RE2_LIBRARY}; ${RE2_ST_INCLUDE_DIR} : ${RE2_ST_LIBRARY}")
|
||||||
|
|||||||
27
cmake/find_snappy.cmake
Normal file
27
cmake/find_snappy.cmake
Normal file
@ -0,0 +1,27 @@
|
|||||||
|
option(USE_INTERNAL_SNAPPY_LIBRARY "Set to FALSE to use system snappy library instead of bundled" ${NOT_UNBUNDLED})
|
||||||
|
|
||||||
|
if(NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/snappy/snappy.h")
|
||||||
|
if(USE_INTERNAL_SNAPPY_LIBRARY)
|
||||||
|
message(WARNING "submodule contrib/snappy is missing. to fix try run: \n git submodule update --init --recursive")
|
||||||
|
set(USE_INTERNAL_SNAPPY_LIBRARY 0)
|
||||||
|
endif()
|
||||||
|
set(MISSING_INTERNAL_SNAPPY_LIBRARY 1)
|
||||||
|
endif()
|
||||||
|
|
||||||
|
if(NOT USE_INTERNAL_SNAPPY_LIBRARY)
|
||||||
|
find_library(SNAPPY_LIBRARY snappy)
|
||||||
|
find_path(SNAPPY_INCLUDE_DIR NAMES snappy.h PATHS ${SNAPPY_INCLUDE_PATHS})
|
||||||
|
endif()
|
||||||
|
|
||||||
|
if(SNAPPY_LIBRARY AND SNAPPY_INCLUDE_DIR)
|
||||||
|
elseif(NOT MISSING_INTERNAL_SNAPPY_LIBRARY)
|
||||||
|
set(SNAPPY_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/contrib/snappy)
|
||||||
|
set(USE_INTERNAL_SNAPPY_LIBRARY 1)
|
||||||
|
set(SNAPPY_LIBRARY snappy)
|
||||||
|
endif()
|
||||||
|
|
||||||
|
if(SNAPPY_LIBRARY AND SNAPPY_INCLUDE_DIR)
|
||||||
|
set(USE_SNAPPY 1)
|
||||||
|
endif()
|
||||||
|
|
||||||
|
message(STATUS "Using snappy=${USE_SNAPPY}: ${SNAPPY_INCLUDE_DIR} : ${SNAPPY_LIBRARY}")
|
||||||
@ -6,7 +6,7 @@ set (OPENSSL_USE_STATIC_LIBS ${USE_STATIC_LIBRARIES})
|
|||||||
|
|
||||||
if (NOT USE_INTERNAL_SSL_LIBRARY)
|
if (NOT USE_INTERNAL_SSL_LIBRARY)
|
||||||
if (APPLE)
|
if (APPLE)
|
||||||
set (OPENSSL_ROOT_DIR "/usr/local/opt/openssl")
|
set (OPENSSL_ROOT_DIR "/usr/local/opt/openssl" CACHE INTERNAL "")
|
||||||
# https://rt.openssl.org/Ticket/Display.html?user=guest&pass=guest&id=2232
|
# https://rt.openssl.org/Ticket/Display.html?user=guest&pass=guest&id=2232
|
||||||
if (USE_STATIC_LIBRARIES)
|
if (USE_STATIC_LIBRARIES)
|
||||||
message(WARNING "Disable USE_STATIC_LIBRARIES if you have linking problems with OpenSSL on MacOS")
|
message(WARNING "Disable USE_STATIC_LIBRARIES if you have linking problems with OpenSSL on MacOS")
|
||||||
|
|||||||
@ -1,10 +1,22 @@
|
|||||||
if (LZ4_INCLUDE_DIR)
|
option (USE_INTERNAL_XXHASH_LIBRARY "Set to FALSE to use system xxHash library instead of bundled" ${NOT_UNBUNDLED})
|
||||||
if (NOT EXISTS "${LZ4_INCLUDE_DIR}/xxhash.h")
|
|
||||||
message (WARNING "LZ4 library does not have XXHash. Support for XXHash will be disabled.")
|
if (USE_INTERNAL_XXHASH_LIBRARY AND NOT USE_INTERNAL_LZ4_LIBRARY)
|
||||||
set (USE_XXHASH 0)
|
message (WARNING "can not use internal xxhash without internal lz4")
|
||||||
else ()
|
set (USE_INTERNAL_XXHASH_LIBRARY 0)
|
||||||
set (USE_XXHASH 1)
|
|
||||||
endif ()
|
|
||||||
endif ()
|
endif ()
|
||||||
|
|
||||||
message (STATUS "Using xxhash=${USE_XXHASH}")
|
if (USE_INTERNAL_XXHASH_LIBRARY)
|
||||||
|
set (XXHASH_LIBRARY lz4)
|
||||||
|
set (XXHASH_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/contrib/lz4/lib)
|
||||||
|
else ()
|
||||||
|
find_library (XXHASH_LIBRARY xxhash)
|
||||||
|
find_path (XXHASH_INCLUDE_DIR NAMES xxhash.h PATHS ${XXHASH_INCLUDE_PATHS})
|
||||||
|
endif ()
|
||||||
|
|
||||||
|
if (XXHASH_LIBRARY AND XXHASH_INCLUDE_DIR)
|
||||||
|
set (USE_XXHASH 1)
|
||||||
|
else ()
|
||||||
|
set (USE_XXHASH 0)
|
||||||
|
endif ()
|
||||||
|
|
||||||
|
message (STATUS "Using xxhash=${USE_XXHASH}: ${XXHASH_INCLUDE_DIR} : ${XXHASH_LIBRARY}")
|
||||||
|
|||||||
@ -14,6 +14,7 @@ if (ZSTD_LIBRARY AND ZSTD_INCLUDE_DIR)
|
|||||||
else ()
|
else ()
|
||||||
set (USE_INTERNAL_ZSTD_LIBRARY 1)
|
set (USE_INTERNAL_ZSTD_LIBRARY 1)
|
||||||
set (ZSTD_LIBRARY zstd)
|
set (ZSTD_LIBRARY zstd)
|
||||||
|
set (ZSTD_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/contrib/zstd/lib)
|
||||||
endif ()
|
endif ()
|
||||||
|
|
||||||
message (STATUS "Using zstd: ${ZSTD_INCLUDE_DIR} : ${ZSTD_LIBRARY}")
|
message (STATUS "Using zstd: ${ZSTD_INCLUDE_DIR} : ${ZSTD_LIBRARY}")
|
||||||
|
|||||||
@ -2,4 +2,5 @@ set(DIVIDE_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/contrib/libdivide)
|
|||||||
set(COMMON_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/libs/libcommon/include ${ClickHouse_BINARY_DIR}/libs/libcommon/include)
|
set(COMMON_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/libs/libcommon/include ${ClickHouse_BINARY_DIR}/libs/libcommon/include)
|
||||||
set(DBMS_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/dbms/src ${ClickHouse_BINARY_DIR}/dbms/src)
|
set(DBMS_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/dbms/src ${ClickHouse_BINARY_DIR}/dbms/src)
|
||||||
set(DOUBLE_CONVERSION_CONTRIB_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/contrib/double-conversion)
|
set(DOUBLE_CONVERSION_CONTRIB_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/contrib/double-conversion)
|
||||||
|
set(METROHASH_CONTRIB_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/contrib/libmetrohash/src)
|
||||||
set(PCG_RANDOM_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/contrib/libpcg-random/include)
|
set(PCG_RANDOM_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/contrib/libpcg-random/include)
|
||||||
|
|||||||
@ -4,6 +4,7 @@
|
|||||||
# include (cmake/limit_jobs.cmake)
|
# include (cmake/limit_jobs.cmake)
|
||||||
|
|
||||||
cmake_host_system_information(RESULT AVAILABLE_PHYSICAL_MEMORY QUERY AVAILABLE_PHYSICAL_MEMORY) # Not available under freebsd
|
cmake_host_system_information(RESULT AVAILABLE_PHYSICAL_MEMORY QUERY AVAILABLE_PHYSICAL_MEMORY) # Not available under freebsd
|
||||||
|
cmake_host_system_information(RESULT NUMBER_OF_LOGICAL_CORES QUERY NUMBER_OF_LOGICAL_CORES)
|
||||||
|
|
||||||
option(PARALLEL_COMPILE_JOBS "Define the maximum number of concurrent compilation jobs" "")
|
option(PARALLEL_COMPILE_JOBS "Define the maximum number of concurrent compilation jobs" "")
|
||||||
if (NOT PARALLEL_COMPILE_JOBS AND AVAILABLE_PHYSICAL_MEMORY AND MAX_COMPILER_MEMORY)
|
if (NOT PARALLEL_COMPILE_JOBS AND AVAILABLE_PHYSICAL_MEMORY AND MAX_COMPILER_MEMORY)
|
||||||
@ -12,7 +13,7 @@ if (NOT PARALLEL_COMPILE_JOBS AND AVAILABLE_PHYSICAL_MEMORY AND MAX_COMPILER_MEM
|
|||||||
set (PARALLEL_COMPILE_JOBS 1)
|
set (PARALLEL_COMPILE_JOBS 1)
|
||||||
endif ()
|
endif ()
|
||||||
endif ()
|
endif ()
|
||||||
if (PARALLEL_COMPILE_JOBS)
|
if (PARALLEL_COMPILE_JOBS AND (NOT NUMBER_OF_LOGICAL_CORES OR PARALLEL_COMPILE_JOBS LESS NUMBER_OF_LOGICAL_CORES))
|
||||||
set(CMAKE_JOB_POOL_COMPILE compile_job_pool${CMAKE_CURRENT_SOURCE_DIR})
|
set(CMAKE_JOB_POOL_COMPILE compile_job_pool${CMAKE_CURRENT_SOURCE_DIR})
|
||||||
string (REGEX REPLACE "[^a-zA-Z0-9]+" "_" CMAKE_JOB_POOL_COMPILE ${CMAKE_JOB_POOL_COMPILE})
|
string (REGEX REPLACE "[^a-zA-Z0-9]+" "_" CMAKE_JOB_POOL_COMPILE ${CMAKE_JOB_POOL_COMPILE})
|
||||||
set_property(GLOBAL APPEND PROPERTY JOB_POOLS ${CMAKE_JOB_POOL_COMPILE}=${PARALLEL_COMPILE_JOBS})
|
set_property(GLOBAL APPEND PROPERTY JOB_POOLS ${CMAKE_JOB_POOL_COMPILE}=${PARALLEL_COMPILE_JOBS})
|
||||||
@ -25,13 +26,12 @@ if (NOT PARALLEL_LINK_JOBS AND AVAILABLE_PHYSICAL_MEMORY AND MAX_LINKER_MEMORY)
|
|||||||
set (PARALLEL_LINK_JOBS 1)
|
set (PARALLEL_LINK_JOBS 1)
|
||||||
endif ()
|
endif ()
|
||||||
endif ()
|
endif ()
|
||||||
if (PARALLEL_COMPILE_JOBS OR PARALLEL_LINK_JOBS)
|
if (PARALLEL_LINK_JOBS AND (NOT NUMBER_OF_LOGICAL_CORES OR PARALLEL_COMPILE_JOBS LESS NUMBER_OF_LOGICAL_CORES))
|
||||||
message(STATUS "${CMAKE_CURRENT_SOURCE_DIR}: Have ${AVAILABLE_PHYSICAL_MEMORY} megabytes of memory. Limiting concurrent linkers jobs to ${PARALLEL_LINK_JOBS} and compiler jobs to ${PARALLEL_COMPILE_JOBS}")
|
|
||||||
endif ()
|
|
||||||
|
|
||||||
if (LLVM_PARALLEL_LINK_JOBS)
|
|
||||||
set(CMAKE_JOB_POOL_LINK link_job_pool${CMAKE_CURRENT_SOURCE_DIR})
|
set(CMAKE_JOB_POOL_LINK link_job_pool${CMAKE_CURRENT_SOURCE_DIR})
|
||||||
string (REGEX REPLACE "[^a-zA-Z0-9]+" "_" CMAKE_JOB_POOL_LINK ${CMAKE_JOB_POOL_LINK})
|
string (REGEX REPLACE "[^a-zA-Z0-9]+" "_" CMAKE_JOB_POOL_LINK ${CMAKE_JOB_POOL_LINK})
|
||||||
set_property(GLOBAL APPEND PROPERTY JOB_POOLS ${CMAKE_JOB_POOL_LINK}=${PARALLEL_LINK_JOBS})
|
set_property(GLOBAL APPEND PROPERTY JOB_POOLS ${CMAKE_JOB_POOL_LINK}=${PARALLEL_LINK_JOBS})
|
||||||
endif ()
|
endif ()
|
||||||
|
|
||||||
|
if (PARALLEL_COMPILE_JOBS OR PARALLEL_LINK_JOBS)
|
||||||
|
message(STATUS "${CMAKE_CURRENT_SOURCE_DIR}: Have ${AVAILABLE_PHYSICAL_MEMORY} megabytes of memory. Limiting concurrent linkers jobs to ${PARALLEL_LINK_JOBS} and compiler jobs to ${PARALLEL_COMPILE_JOBS}")
|
||||||
|
endif ()
|
||||||
|
|||||||
@ -13,6 +13,9 @@ list(APPEND dirs ${dirs1})
|
|||||||
get_property (dirs1 TARGET cityhash PROPERTY INCLUDE_DIRECTORIES)
|
get_property (dirs1 TARGET cityhash PROPERTY INCLUDE_DIRECTORIES)
|
||||||
list(APPEND dirs ${dirs1})
|
list(APPEND dirs ${dirs1})
|
||||||
|
|
||||||
|
get_property (dirs1 TARGET roaring PROPERTY INCLUDE_DIRECTORIES)
|
||||||
|
list(APPEND dirs ${dirs1})
|
||||||
|
|
||||||
if (USE_INTERNAL_BOOST_LIBRARY)
|
if (USE_INTERNAL_BOOST_LIBRARY)
|
||||||
get_property (dirs1 TARGET ${Boost_PROGRAM_OPTIONS_LIBRARY} PROPERTY INCLUDE_DIRECTORIES)
|
get_property (dirs1 TARGET ${Boost_PROGRAM_OPTIONS_LIBRARY} PROPERTY INCLUDE_DIRECTORIES)
|
||||||
list(APPEND dirs ${dirs1})
|
list(APPEND dirs ${dirs1})
|
||||||
|
|||||||
@ -27,6 +27,9 @@ if (HAVE_SSE41)
|
|||||||
set (COMPILER_FLAGS "${COMPILER_FLAGS} ${TEST_FLAG}")
|
set (COMPILER_FLAGS "${COMPILER_FLAGS} ${TEST_FLAG}")
|
||||||
endif ()
|
endif ()
|
||||||
|
|
||||||
|
if (ARCH_PPC64LE)
|
||||||
|
set (COMPILER_FLAGS "${COMPILER_FLAGS} -maltivec -D__SSE2__=1 -DNO_WARN_X86_INTRINSICS")
|
||||||
|
endif ()
|
||||||
|
|
||||||
# gcc -dM -E -msse4.2 - < /dev/null | sort > gcc-dump-sse42
|
# gcc -dM -E -msse4.2 - < /dev/null | sort > gcc-dump-sse42
|
||||||
#define __SSE4_2__ 1
|
#define __SSE4_2__ 1
|
||||||
|
|||||||
107
contrib/CMakeLists.txt
vendored
107
contrib/CMakeLists.txt
vendored
@ -1,13 +1,15 @@
|
|||||||
# Third-party libraries may have substandard code.
|
# Third-party libraries may have substandard code.
|
||||||
|
|
||||||
if (CMAKE_CXX_COMPILER_ID STREQUAL "GNU")
|
if (CMAKE_CXX_COMPILER_ID STREQUAL "GNU")
|
||||||
set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wno-unused-function -Wno-unused-variable -Wno-unused-but-set-variable -Wno-unused-result -Wno-deprecated-declarations -Wno-maybe-uninitialized -Wno-format -Wno-misleading-indentation -Wno-stringop-overflow")
|
set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wno-unused-function -Wno-unused-variable -Wno-unused-but-set-variable -Wno-unused-result -Wno-deprecated-declarations -Wno-maybe-uninitialized -Wno-format -Wno-misleading-indentation -Wno-stringop-overflow -Wno-implicit-function-declaration -Wno-return-type -Wno-array-bounds -Wno-bool-compare -Wno-int-conversion -Wno-switch")
|
||||||
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-old-style-cast -Wno-unused-function -Wno-unused-variable -Wno-unused-but-set-variable -Wno-unused-result -Wno-deprecated-declarations -Wno-non-virtual-dtor -Wno-maybe-uninitialized -Wno-format -Wno-misleading-indentation -Wno-implicit-fallthrough -Wno-class-memaccess -Wno-sign-compare -std=c++1z")
|
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-old-style-cast -Wno-unused-function -Wno-unused-variable -Wno-unused-but-set-variable -Wno-unused-result -Wno-deprecated-declarations -Wno-non-virtual-dtor -Wno-maybe-uninitialized -Wno-format -Wno-misleading-indentation -Wno-implicit-fallthrough -Wno-class-memaccess -Wno-sign-compare -std=c++1z")
|
||||||
elseif (CMAKE_CXX_COMPILER_ID STREQUAL "Clang")
|
elseif (CMAKE_CXX_COMPILER_ID STREQUAL "Clang")
|
||||||
set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wno-unused-function -Wno-unused-variable -Wno-unused-result -Wno-deprecated-declarations -Wno-format -Wno-parentheses-equality -Wno-tautological-constant-compare -Wno-tautological-constant-out-of-range-compare -Wno-implicit-function-declaration -Wno-return-type -Wno-pointer-bool-conversion -Wno-enum-conversion -Wno-int-conversion -Wno-switch")
|
set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wno-unused-function -Wno-unused-variable -Wno-unused-result -Wno-deprecated-declarations -Wno-format -Wno-parentheses-equality -Wno-tautological-constant-compare -Wno-tautological-constant-out-of-range-compare -Wno-implicit-function-declaration -Wno-return-type -Wno-pointer-bool-conversion -Wno-enum-conversion -Wno-int-conversion -Wno-switch")
|
||||||
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-old-style-cast -Wno-unused-function -Wno-unused-variable -Wno-unused-result -Wno-deprecated-declarations -Wno-non-virtual-dtor -Wno-format -Wno-inconsistent-missing-override -std=c++1z")
|
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-old-style-cast -Wno-unused-function -Wno-unused-variable -Wno-unused-result -Wno-deprecated-declarations -Wno-non-virtual-dtor -Wno-format -Wno-inconsistent-missing-override -std=c++1z")
|
||||||
endif ()
|
endif ()
|
||||||
|
|
||||||
|
set_property(DIRECTORY PROPERTY EXCLUDE_FROM_ALL 1)
|
||||||
|
|
||||||
if (USE_INTERNAL_BOOST_LIBRARY)
|
if (USE_INTERNAL_BOOST_LIBRARY)
|
||||||
add_subdirectory (boost-cmake)
|
add_subdirectory (boost-cmake)
|
||||||
endif ()
|
endif ()
|
||||||
@ -44,6 +46,7 @@ if (USE_INTERNAL_METROHASH_LIBRARY)
|
|||||||
endif ()
|
endif ()
|
||||||
|
|
||||||
add_subdirectory (murmurhash)
|
add_subdirectory (murmurhash)
|
||||||
|
add_subdirectory (croaring)
|
||||||
|
|
||||||
if (USE_INTERNAL_BTRIE_LIBRARY)
|
if (USE_INTERNAL_BTRIE_LIBRARY)
|
||||||
add_subdirectory (libbtrie)
|
add_subdirectory (libbtrie)
|
||||||
@ -54,6 +57,8 @@ if (USE_INTERNAL_UNWIND_LIBRARY)
|
|||||||
endif ()
|
endif ()
|
||||||
|
|
||||||
if (USE_INTERNAL_ZLIB_LIBRARY)
|
if (USE_INTERNAL_ZLIB_LIBRARY)
|
||||||
|
set (ZLIB_ENABLE_TESTS 0 CACHE INTERNAL "")
|
||||||
|
set (SKIP_INSTALL_ALL 1 CACHE INTERNAL "")
|
||||||
set (ZLIB_COMPAT 1 CACHE INTERNAL "") # also enables WITH_GZFILEOP
|
set (ZLIB_COMPAT 1 CACHE INTERNAL "") # also enables WITH_GZFILEOP
|
||||||
set (WITH_NATIVE_INSTRUCTIONS ${ARCH_NATIVE} CACHE INTERNAL "")
|
set (WITH_NATIVE_INSTRUCTIONS ${ARCH_NATIVE} CACHE INTERNAL "")
|
||||||
if (OS_FREEBSD OR ARCH_I386)
|
if (OS_FREEBSD OR ARCH_I386)
|
||||||
@ -74,15 +79,15 @@ if (USE_INTERNAL_ZLIB_LIBRARY)
|
|||||||
target_compile_definitions (zlibstatic PUBLIC X86_64 UNALIGNED_OK)
|
target_compile_definitions (zlibstatic PUBLIC X86_64 UNALIGNED_OK)
|
||||||
endif ()
|
endif ()
|
||||||
|
|
||||||
set_target_properties(example PROPERTIES EXCLUDE_FROM_ALL 1)
|
#set_target_properties(example PROPERTIES EXCLUDE_FROM_ALL 1)
|
||||||
if (TARGET example64)
|
#if (TARGET example64)
|
||||||
set_target_properties(example64 PROPERTIES EXCLUDE_FROM_ALL 1)
|
# set_target_properties(example64 PROPERTIES EXCLUDE_FROM_ALL 1)
|
||||||
endif ()
|
#endif ()
|
||||||
|
|
||||||
set_target_properties(minigzip PROPERTIES EXCLUDE_FROM_ALL 1)
|
#set_target_properties(minigzip PROPERTIES EXCLUDE_FROM_ALL 1)
|
||||||
if (TARGET minigzip64)
|
#if (TARGET minigzip64)
|
||||||
set_target_properties(minigzip64 PROPERTIES EXCLUDE_FROM_ALL 1)
|
# set_target_properties(minigzip64 PROPERTIES EXCLUDE_FROM_ALL 1)
|
||||||
endif ()
|
#endif ()
|
||||||
endif ()
|
endif ()
|
||||||
|
|
||||||
if (USE_INTERNAL_CCTZ_LIBRARY)
|
if (USE_INTERNAL_CCTZ_LIBRARY)
|
||||||
@ -105,8 +110,12 @@ if (USE_INTERNAL_SSL_LIBRARY)
|
|||||||
if (NOT MAKE_STATIC_LIBRARIES)
|
if (NOT MAKE_STATIC_LIBRARIES)
|
||||||
set (BUILD_SHARED 1)
|
set (BUILD_SHARED 1)
|
||||||
endif ()
|
endif ()
|
||||||
set (USE_SHARED ${USE_STATIC_LIBRARIES})
|
|
||||||
set (LIBRESSL_SKIP_INSTALL 1)
|
# By default, ${CMAKE_INSTALL_PREFIX}/etc/ssl is selected - that is not what we need.
|
||||||
|
# We need to use system wide ssl directory.
|
||||||
|
set (OPENSSLDIR "/etc/ssl")
|
||||||
|
|
||||||
|
set (LIBRESSL_SKIP_INSTALL 1 CACHE INTERNAL "")
|
||||||
add_subdirectory (ssl)
|
add_subdirectory (ssl)
|
||||||
target_include_directories(${OPENSSL_CRYPTO_LIBRARY} SYSTEM PUBLIC ${OPENSSL_INCLUDE_DIR})
|
target_include_directories(${OPENSSL_CRYPTO_LIBRARY} SYSTEM PUBLIC ${OPENSSL_INCLUDE_DIR})
|
||||||
target_include_directories(${OPENSSL_SSL_LIBRARY} SYSTEM PUBLIC ${OPENSSL_INCLUDE_DIR})
|
target_include_directories(${OPENSSL_SSL_LIBRARY} SYSTEM PUBLIC ${OPENSSL_INCLUDE_DIR})
|
||||||
@ -125,6 +134,10 @@ if (USE_INTERNAL_RDKAFKA_LIBRARY)
|
|||||||
target_include_directories(rdkafka BEFORE PRIVATE ${OPENSSL_INCLUDE_DIR})
|
target_include_directories(rdkafka BEFORE PRIVATE ${OPENSSL_INCLUDE_DIR})
|
||||||
endif ()
|
endif ()
|
||||||
|
|
||||||
|
if (USE_RDKAFKA)
|
||||||
|
add_subdirectory (cppkafka-cmake)
|
||||||
|
endif()
|
||||||
|
|
||||||
if (ENABLE_ODBC AND USE_INTERNAL_ODBC_LIBRARY)
|
if (ENABLE_ODBC AND USE_INTERNAL_ODBC_LIBRARY)
|
||||||
add_subdirectory (unixodbc-cmake)
|
add_subdirectory (unixodbc-cmake)
|
||||||
endif ()
|
endif ()
|
||||||
@ -138,7 +151,68 @@ if (ENABLE_CAPNP AND USE_INTERNAL_CAPNP_LIBRARY)
|
|||||||
target_include_directories(${CAPNP_LIBRARY} PUBLIC $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/capnproto/c++/src>)
|
target_include_directories(${CAPNP_LIBRARY} PUBLIC $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/capnproto/c++/src>)
|
||||||
endif ()
|
endif ()
|
||||||
|
|
||||||
|
if (USE_INTERNAL_PARQUET_LIBRARY)
|
||||||
|
if (USE_INTERNAL_PARQUET_LIBRARY_NATIVE_CMAKE)
|
||||||
|
# We dont use arrow's cmakefiles because they uses too many depends and download some libs in compile time
|
||||||
|
# But this mode can be used for updating auto-generated parquet files:
|
||||||
|
# cmake -DUSE_INTERNAL_PARQUET_LIBRARY_NATIVE_CMAKE=1 -DUSE_STATIC_LIBRARIES=0
|
||||||
|
# copy {BUILD_DIR}/contrib/arrow/cpp/src/parquet/*.cpp,*.h -> /contrib/arrow-cmake/cpp/src/parquet/
|
||||||
|
|
||||||
|
# Also useful parquet reader:
|
||||||
|
# cd contrib/arrow/cpp/build && mkdir -p build && cmake .. -DPARQUET_BUILD_EXECUTABLES=1 && make -j8
|
||||||
|
# contrib/arrow/cpp/build/debug/parquet-reader some_file.parquet
|
||||||
|
|
||||||
|
set (ARROW_COMPUTE ON CACHE INTERNAL "")
|
||||||
|
set (ARROW_PARQUET ON CACHE INTERNAL "")
|
||||||
|
set (ARROW_VERBOSE_THIRDPARTY_BUILD ON CACHE INTERNAL "")
|
||||||
|
set (ARROW_BUILD_SHARED 1 CACHE INTERNAL "")
|
||||||
|
set (ARROW_BOOST_HEADER_ONLY ON CACHE INTERNAL "")
|
||||||
|
#set (BOOST_INCLUDEDIR Boost_INCLUDE_DIRS)
|
||||||
|
set (Boost_FOUND 1 CACHE INTERNAL "")
|
||||||
|
#set (ZLIB_HOME ${ZLIB_INCLUDE_DIR})
|
||||||
|
#set (ZLIB_FOUND 1)
|
||||||
|
if (MAKE_STATIC_LIBRARIES)
|
||||||
|
set (PARQUET_ARROW_LINKAGE "static" CACHE INTERNAL "")
|
||||||
|
set (ARROW_TEST_LINKAGE "static" CACHE INTERNAL "")
|
||||||
|
set (ARROW_BUILD_STATIC ${MAKE_STATIC_LIBRARIES} CACHE INTERNAL "")
|
||||||
|
else()
|
||||||
|
set (PARQUET_ARROW_LINKAGE "shared" CACHE INTERNAL "")
|
||||||
|
set (ARROW_TEST_LINKAGE "shared" CACHE INTERNAL "")
|
||||||
|
endif()
|
||||||
|
|
||||||
|
if(CMAKE_BUILD_TYPE STREQUAL "RELWITHDEBINFO")
|
||||||
|
set(_save_build_type ${CMAKE_BUILD_TYPE})
|
||||||
|
set(CMAKE_BUILD_TYPE RELEASE)
|
||||||
|
endif()
|
||||||
|
|
||||||
|
# Because Arrow uses CMAKE_SOURCE_DIR as a project path
|
||||||
|
# Hopefully will be fixed in https://github.com/apache/arrow/pull/2676
|
||||||
|
set (CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} "${ClickHouse_SOURCE_DIR}/contrib/arrow/cpp/cmake_modules")
|
||||||
|
add_subdirectory (arrow/cpp)
|
||||||
|
|
||||||
|
if(_save_build_type)
|
||||||
|
set(CMAKE_BUILD_TYPE ${_save_build_type})
|
||||||
|
endif()
|
||||||
|
|
||||||
|
else()
|
||||||
|
|
||||||
|
if(USE_INTERNAL_SNAPPY_LIBRARY)
|
||||||
|
set(SNAPPY_BUILD_TESTS 0 CACHE INTERNAL "")
|
||||||
|
if (NOT MAKE_STATIC_LIBRARIES)
|
||||||
|
set(BUILD_SHARED_LIBS 1) # TODO: set at root dir
|
||||||
|
endif()
|
||||||
|
add_subdirectory(snappy)
|
||||||
|
if(SANITIZE STREQUAL "undefined")
|
||||||
|
target_compile_options(${SNAPPY_LIBRARY} PRIVATE -fno-sanitize=undefined)
|
||||||
|
endif()
|
||||||
|
endif()
|
||||||
|
|
||||||
|
add_subdirectory(arrow-cmake)
|
||||||
|
endif()
|
||||||
|
endif()
|
||||||
|
|
||||||
if (USE_INTERNAL_POCO_LIBRARY)
|
if (USE_INTERNAL_POCO_LIBRARY)
|
||||||
|
set (POCO_VERBOSE_MESSAGES 0 CACHE INTERNAL "")
|
||||||
set (save_CMAKE_CXX_FLAGS ${CMAKE_CXX_FLAGS})
|
set (save_CMAKE_CXX_FLAGS ${CMAKE_CXX_FLAGS})
|
||||||
set (save_CMAKE_C_FLAGS ${CMAKE_C_FLAGS})
|
set (save_CMAKE_C_FLAGS ${CMAKE_C_FLAGS})
|
||||||
set (_save ${ENABLE_TESTS})
|
set (_save ${ENABLE_TESTS})
|
||||||
@ -166,6 +240,9 @@ if (USE_INTERNAL_GTEST_LIBRARY)
|
|||||||
# avoid problems with <regexp.h>
|
# avoid problems with <regexp.h>
|
||||||
target_compile_definitions (gtest INTERFACE GTEST_HAS_POSIX_RE=0)
|
target_compile_definitions (gtest INTERFACE GTEST_HAS_POSIX_RE=0)
|
||||||
target_include_directories (gtest SYSTEM INTERFACE ${ClickHouse_SOURCE_DIR}/contrib/googletest/include)
|
target_include_directories (gtest SYSTEM INTERFACE ${ClickHouse_SOURCE_DIR}/contrib/googletest/include)
|
||||||
|
elseif(GTEST_SRC_DIR)
|
||||||
|
add_subdirectory(${GTEST_SRC_DIR}/googletest ${CMAKE_CURRENT_BINARY_DIR}/googletest)
|
||||||
|
target_compile_definitions(gtest INTERFACE GTEST_HAS_POSIX_RE=0)
|
||||||
endif()
|
endif()
|
||||||
|
|
||||||
if (USE_INTERNAL_LLVM_LIBRARY)
|
if (USE_INTERNAL_LLVM_LIBRARY)
|
||||||
@ -228,14 +305,18 @@ if (USE_INTERNAL_LIBXML2_LIBRARY)
|
|||||||
add_subdirectory(libxml2-cmake)
|
add_subdirectory(libxml2-cmake)
|
||||||
endif ()
|
endif ()
|
||||||
|
|
||||||
if (USE_INTERNAL_HDFS3_LIBRARY)
|
if (USE_INTERNAL_BROTLI_LIBRARY)
|
||||||
include(${ClickHouse_SOURCE_DIR}/cmake/find_protobuf.cmake)
|
add_subdirectory(brotli-cmake)
|
||||||
|
endif ()
|
||||||
|
|
||||||
if (USE_INTERNAL_PROTOBUF_LIBRARY)
|
if (USE_INTERNAL_PROTOBUF_LIBRARY)
|
||||||
set(protobuf_BUILD_TESTS OFF CACHE INTERNAL "" FORCE)
|
set(protobuf_BUILD_TESTS OFF CACHE INTERNAL "" FORCE)
|
||||||
set(protobuf_BUILD_SHARED_LIBS OFF CACHE INTERNAL "" FORCE)
|
set(protobuf_BUILD_SHARED_LIBS OFF CACHE INTERNAL "" FORCE)
|
||||||
set(protobuf_WITH_ZLIB 0 CACHE INTERNAL "" FORCE) # actually will use zlib, but skip find
|
set(protobuf_WITH_ZLIB 0 CACHE INTERNAL "" FORCE) # actually will use zlib, but skip find
|
||||||
add_subdirectory(protobuf/cmake)
|
add_subdirectory(protobuf/cmake)
|
||||||
endif ()
|
endif ()
|
||||||
|
|
||||||
|
if (USE_INTERNAL_HDFS3_LIBRARY)
|
||||||
add_subdirectory(libhdfs3-cmake)
|
add_subdirectory(libhdfs3-cmake)
|
||||||
endif ()
|
endif ()
|
||||||
|
|
||||||
|
|||||||
1
contrib/arrow
vendored
Submodule
1
contrib/arrow
vendored
Submodule
@ -0,0 +1 @@
|
|||||||
|
Subproject commit 87ac6fddaf21d0b4ee8b8090533ff293db0da1b4
|
||||||
216
contrib/arrow-cmake/CMakeLists.txt
Normal file
216
contrib/arrow-cmake/CMakeLists.txt
Normal file
@ -0,0 +1,216 @@
|
|||||||
|
# === thrift
|
||||||
|
|
||||||
|
set(LIBRARY_DIR ${ClickHouse_SOURCE_DIR}/contrib/thrift/lib/cpp)
|
||||||
|
# contrib/thrift/lib/cpp/CMakeLists.txt
|
||||||
|
set(thriftcpp_SOURCES
|
||||||
|
${LIBRARY_DIR}/src/thrift/TApplicationException.cpp
|
||||||
|
${LIBRARY_DIR}/src/thrift/TOutput.cpp
|
||||||
|
${LIBRARY_DIR}/src/thrift/async/TAsyncChannel.cpp
|
||||||
|
${LIBRARY_DIR}/src/thrift/async/TAsyncProtocolProcessor.cpp
|
||||||
|
${LIBRARY_DIR}/src/thrift/async/TConcurrentClientSyncInfo.h
|
||||||
|
${LIBRARY_DIR}/src/thrift/async/TConcurrentClientSyncInfo.cpp
|
||||||
|
${LIBRARY_DIR}/src/thrift/concurrency/ThreadManager.cpp
|
||||||
|
${LIBRARY_DIR}/src/thrift/concurrency/TimerManager.cpp
|
||||||
|
${LIBRARY_DIR}/src/thrift/concurrency/Util.cpp
|
||||||
|
${LIBRARY_DIR}/src/thrift/processor/PeekProcessor.cpp
|
||||||
|
${LIBRARY_DIR}/src/thrift/protocol/TBase64Utils.cpp
|
||||||
|
${LIBRARY_DIR}/src/thrift/protocol/TDebugProtocol.cpp
|
||||||
|
${LIBRARY_DIR}/src/thrift/protocol/TJSONProtocol.cpp
|
||||||
|
${LIBRARY_DIR}/src/thrift/protocol/TMultiplexedProtocol.cpp
|
||||||
|
${LIBRARY_DIR}/src/thrift/protocol/TProtocol.cpp
|
||||||
|
${LIBRARY_DIR}/src/thrift/transport/TTransportException.cpp
|
||||||
|
${LIBRARY_DIR}/src/thrift/transport/TFDTransport.cpp
|
||||||
|
${LIBRARY_DIR}/src/thrift/transport/TSimpleFileTransport.cpp
|
||||||
|
${LIBRARY_DIR}/src/thrift/transport/THttpTransport.cpp
|
||||||
|
${LIBRARY_DIR}/src/thrift/transport/THttpClient.cpp
|
||||||
|
${LIBRARY_DIR}/src/thrift/transport/THttpServer.cpp
|
||||||
|
${LIBRARY_DIR}/src/thrift/transport/TSocket.cpp
|
||||||
|
${LIBRARY_DIR}/src/thrift/transport/TSocketPool.cpp
|
||||||
|
${LIBRARY_DIR}/src/thrift/transport/TServerSocket.cpp
|
||||||
|
${LIBRARY_DIR}/src/thrift/transport/TTransportUtils.cpp
|
||||||
|
${LIBRARY_DIR}/src/thrift/transport/TBufferTransports.cpp
|
||||||
|
${LIBRARY_DIR}/src/thrift/server/TConnectedClient.cpp
|
||||||
|
${LIBRARY_DIR}/src/thrift/server/TServerFramework.cpp
|
||||||
|
${LIBRARY_DIR}/src/thrift/server/TSimpleServer.cpp
|
||||||
|
${LIBRARY_DIR}/src/thrift/server/TThreadPoolServer.cpp
|
||||||
|
${LIBRARY_DIR}/src/thrift/server/TThreadedServer.cpp
|
||||||
|
)
|
||||||
|
set( thriftcpp_threads_SOURCES
|
||||||
|
${LIBRARY_DIR}/src/thrift/concurrency/ThreadFactory.cpp
|
||||||
|
${LIBRARY_DIR}/src/thrift/concurrency/Thread.cpp
|
||||||
|
${LIBRARY_DIR}/src/thrift/concurrency/Monitor.cpp
|
||||||
|
${LIBRARY_DIR}/src/thrift/concurrency/Mutex.cpp
|
||||||
|
)
|
||||||
|
add_library(${THRIFT_LIBRARY} ${LINK_MODE} ${thriftcpp_SOURCES} ${thriftcpp_threads_SOURCES})
|
||||||
|
set_target_properties(${THRIFT_LIBRARY} PROPERTIES CXX_STANDARD 14) # REMOVE after https://github.com/apache/thrift/pull/1641
|
||||||
|
target_include_directories(${THRIFT_LIBRARY} SYSTEM PUBLIC ${ClickHouse_SOURCE_DIR}/contrib/thrift/lib/cpp/src PRIVATE ${Boost_INCLUDE_DIRS})
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
# === arrow
|
||||||
|
|
||||||
|
set(LIBRARY_DIR ${ClickHouse_SOURCE_DIR}/contrib/arrow/cpp/src/arrow)
|
||||||
|
# arrow/cpp/src/arrow/CMakeLists.txt
|
||||||
|
set(ARROW_SRCS
|
||||||
|
${LIBRARY_DIR}/array.cc
|
||||||
|
|
||||||
|
${LIBRARY_DIR}/builder.cc
|
||||||
|
${LIBRARY_DIR}/array/builder_adaptive.cc
|
||||||
|
${LIBRARY_DIR}/array/builder_base.cc
|
||||||
|
${LIBRARY_DIR}/array/builder_binary.cc
|
||||||
|
${LIBRARY_DIR}/array/builder_decimal.cc
|
||||||
|
${LIBRARY_DIR}/array/builder_dict.cc
|
||||||
|
${LIBRARY_DIR}/array/builder_nested.cc
|
||||||
|
${LIBRARY_DIR}/array/builder_primitive.cc
|
||||||
|
|
||||||
|
${LIBRARY_DIR}/buffer.cc
|
||||||
|
${LIBRARY_DIR}/compare.cc
|
||||||
|
${LIBRARY_DIR}/memory_pool.cc
|
||||||
|
${LIBRARY_DIR}/pretty_print.cc
|
||||||
|
${LIBRARY_DIR}/record_batch.cc
|
||||||
|
${LIBRARY_DIR}/status.cc
|
||||||
|
${LIBRARY_DIR}/table.cc
|
||||||
|
${LIBRARY_DIR}/table_builder.cc
|
||||||
|
${LIBRARY_DIR}/tensor.cc
|
||||||
|
${LIBRARY_DIR}/sparse_tensor.cc
|
||||||
|
${LIBRARY_DIR}/type.cc
|
||||||
|
${LIBRARY_DIR}/visitor.cc
|
||||||
|
|
||||||
|
${LIBRARY_DIR}/csv/converter.cc
|
||||||
|
${LIBRARY_DIR}/csv/chunker.cc
|
||||||
|
${LIBRARY_DIR}/csv/column-builder.cc
|
||||||
|
${LIBRARY_DIR}/csv/options.cc
|
||||||
|
${LIBRARY_DIR}/csv/parser.cc
|
||||||
|
${LIBRARY_DIR}/csv/reader.cc
|
||||||
|
|
||||||
|
${LIBRARY_DIR}/io/buffered.cc
|
||||||
|
${LIBRARY_DIR}/io/compressed.cc
|
||||||
|
${LIBRARY_DIR}/io/file.cc
|
||||||
|
${LIBRARY_DIR}/io/interfaces.cc
|
||||||
|
${LIBRARY_DIR}/io/memory.cc
|
||||||
|
${LIBRARY_DIR}/io/readahead.cc
|
||||||
|
|
||||||
|
${LIBRARY_DIR}/util/bit-util.cc
|
||||||
|
${LIBRARY_DIR}/util/compression.cc
|
||||||
|
${LIBRARY_DIR}/util/cpu-info.cc
|
||||||
|
${LIBRARY_DIR}/util/decimal.cc
|
||||||
|
${LIBRARY_DIR}/util/int-util.cc
|
||||||
|
${LIBRARY_DIR}/util/io-util.cc
|
||||||
|
${LIBRARY_DIR}/util/logging.cc
|
||||||
|
${LIBRARY_DIR}/util/key_value_metadata.cc
|
||||||
|
${LIBRARY_DIR}/util/task-group.cc
|
||||||
|
${LIBRARY_DIR}/util/thread-pool.cc
|
||||||
|
${LIBRARY_DIR}/util/trie.cc
|
||||||
|
${LIBRARY_DIR}/util/utf8.cc
|
||||||
|
)
|
||||||
|
|
||||||
|
set(ARROW_SRCS ${ARROW_SRCS}
|
||||||
|
${LIBRARY_DIR}/compute/context.cc
|
||||||
|
${LIBRARY_DIR}/compute/kernels/boolean.cc
|
||||||
|
${LIBRARY_DIR}/compute/kernels/cast.cc
|
||||||
|
${LIBRARY_DIR}/compute/kernels/hash.cc
|
||||||
|
${LIBRARY_DIR}/compute/kernels/util-internal.cc
|
||||||
|
)
|
||||||
|
|
||||||
|
if (LZ4_INCLUDE_DIR AND LZ4_LIBRARY)
|
||||||
|
set(ARROW_WITH_LZ4 1)
|
||||||
|
endif()
|
||||||
|
|
||||||
|
if(SNAPPY_INCLUDE_DIR AND SNAPPY_LIBRARY)
|
||||||
|
set(ARROW_WITH_SNAPPY 1)
|
||||||
|
endif()
|
||||||
|
|
||||||
|
if(ZLIB_INCLUDE_DIR AND ZLIB_LIBRARIES)
|
||||||
|
set(ARROW_WITH_ZLIB 1)
|
||||||
|
endif()
|
||||||
|
|
||||||
|
if (ZSTD_INCLUDE_DIR AND ZSTD_LIBRARY)
|
||||||
|
set(ARROW_WITH_ZSTD 1)
|
||||||
|
endif()
|
||||||
|
|
||||||
|
if (ARROW_WITH_LZ4)
|
||||||
|
add_definitions(-DARROW_WITH_LZ4)
|
||||||
|
SET(ARROW_SRCS ${LIBRARY_DIR}/util/compression_lz4.cc ${ARROW_SRCS})
|
||||||
|
endif()
|
||||||
|
|
||||||
|
if (ARROW_WITH_SNAPPY)
|
||||||
|
add_definitions(-DARROW_WITH_SNAPPY)
|
||||||
|
SET(ARROW_SRCS ${LIBRARY_DIR}/util/compression_snappy.cc ${ARROW_SRCS})
|
||||||
|
endif()
|
||||||
|
|
||||||
|
if (ARROW_WITH_ZLIB)
|
||||||
|
add_definitions(-DARROW_WITH_ZLIB)
|
||||||
|
SET(ARROW_SRCS ${LIBRARY_DIR}/util/compression_zlib.cc ${ARROW_SRCS})
|
||||||
|
endif()
|
||||||
|
|
||||||
|
if (ARROW_WITH_ZSTD)
|
||||||
|
add_definitions(-DARROW_WITH_ZSTD)
|
||||||
|
SET(ARROW_SRCS ${LIBRARY_DIR}/util/compression_zstd.cc ${ARROW_SRCS})
|
||||||
|
endif()
|
||||||
|
|
||||||
|
|
||||||
|
add_library(${ARROW_LIBRARY} ${LINK_MODE} ${ARROW_SRCS})
|
||||||
|
target_include_directories(${ARROW_LIBRARY} SYSTEM PUBLIC ${ClickHouse_SOURCE_DIR}/contrib/arrow/cpp/src PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/cpp/src ${Boost_INCLUDE_DIRS})
|
||||||
|
target_link_libraries(${ARROW_LIBRARY} PRIVATE ${DOUBLE_CONVERSION_LIBRARIES} Threads::Threads)
|
||||||
|
if (ARROW_WITH_LZ4)
|
||||||
|
target_link_libraries(${ARROW_LIBRARY} PRIVATE ${LZ4_LIBRARY})
|
||||||
|
endif()
|
||||||
|
if (ARROW_WITH_SNAPPY)
|
||||||
|
target_link_libraries(${ARROW_LIBRARY} PRIVATE ${SNAPPY_LIBRARY})
|
||||||
|
endif()
|
||||||
|
if (ARROW_WITH_ZLIB)
|
||||||
|
target_link_libraries(${ARROW_LIBRARY} PRIVATE ${ZLIB_LIBRARIES})
|
||||||
|
endif()
|
||||||
|
if (ARROW_WITH_ZSTD)
|
||||||
|
target_link_libraries(${ARROW_LIBRARY} PRIVATE ${ZSTD_LIBRARY})
|
||||||
|
endif()
|
||||||
|
|
||||||
|
|
||||||
|
# === parquet
|
||||||
|
|
||||||
|
set(LIBRARY_DIR ${ClickHouse_SOURCE_DIR}/contrib/arrow/cpp/src/parquet)
|
||||||
|
# arrow/cpp/src/parquet/CMakeLists.txt
|
||||||
|
set(PARQUET_SRCS
|
||||||
|
${LIBRARY_DIR}/arrow/reader.cc
|
||||||
|
${LIBRARY_DIR}/arrow/record_reader.cc
|
||||||
|
${LIBRARY_DIR}/arrow/schema.cc
|
||||||
|
${LIBRARY_DIR}/arrow/writer.cc
|
||||||
|
${LIBRARY_DIR}/bloom_filter.cc
|
||||||
|
${LIBRARY_DIR}/column_reader.cc
|
||||||
|
${LIBRARY_DIR}/column_scanner.cc
|
||||||
|
${LIBRARY_DIR}/column_writer.cc
|
||||||
|
${LIBRARY_DIR}/file_reader.cc
|
||||||
|
${LIBRARY_DIR}/file_writer.cc
|
||||||
|
${LIBRARY_DIR}/metadata.cc
|
||||||
|
${LIBRARY_DIR}/murmur3.cc
|
||||||
|
${LIBRARY_DIR}/printer.cc
|
||||||
|
${LIBRARY_DIR}/schema.cc
|
||||||
|
${LIBRARY_DIR}/statistics.cc
|
||||||
|
${LIBRARY_DIR}/types.cc
|
||||||
|
${LIBRARY_DIR}/util/comparison.cc
|
||||||
|
${LIBRARY_DIR}/util/memory.cc
|
||||||
|
)
|
||||||
|
#list(TRANSFORM PARQUET_SRCS PREPEND ${LIBRARY_DIR}/) # cmake 3.12
|
||||||
|
list(APPEND PARQUET_SRCS
|
||||||
|
${CMAKE_CURRENT_SOURCE_DIR}/cpp/src/parquet/parquet_constants.cpp
|
||||||
|
${CMAKE_CURRENT_SOURCE_DIR}/cpp/src/parquet/parquet_types.cpp
|
||||||
|
)
|
||||||
|
add_library(${PARQUET_LIBRARY} ${LINK_MODE} ${PARQUET_SRCS})
|
||||||
|
target_include_directories(${PARQUET_LIBRARY} SYSTEM PUBLIC ${ClickHouse_SOURCE_DIR}/contrib/arrow/cpp/src ${CMAKE_CURRENT_SOURCE_DIR}/cpp/src)
|
||||||
|
include(${ClickHouse_SOURCE_DIR}/contrib/thrift/build/cmake/ConfigureChecks.cmake) # makes config.h
|
||||||
|
target_link_libraries(${PARQUET_LIBRARY} PUBLIC ${ARROW_LIBRARY} PRIVATE ${THRIFT_LIBRARY} ${Boost_REGEX_LIBRARY})
|
||||||
|
target_include_directories(${PARQUET_LIBRARY} PRIVATE ${Boost_INCLUDE_DIRS})
|
||||||
|
|
||||||
|
if(SANITIZE STREQUAL "undefined")
|
||||||
|
target_compile_options(${PARQUET_LIBRARY} PRIVATE -fno-sanitize=undefined)
|
||||||
|
target_compile_options(${ARROW_LIBRARY} PRIVATE -fno-sanitize=undefined)
|
||||||
|
endif()
|
||||||
|
|
||||||
|
# === tools
|
||||||
|
|
||||||
|
set(TOOLS_DIR ${ClickHouse_SOURCE_DIR}/contrib/arrow/cpp/tools/parquet)
|
||||||
|
set(PARQUET_TOOLS parquet-dump-schema parquet-reader parquet-scan)
|
||||||
|
foreach(TOOL ${PARQUET_TOOLS})
|
||||||
|
add_executable(${TOOL} ${TOOLS_DIR}/${TOOL}.cc)
|
||||||
|
target_link_libraries(${TOOL} PRIVATE ${PARQUET_LIBRARY})
|
||||||
|
endforeach()
|
||||||
1
contrib/arrow-cmake/build/cmake/config.h.in
Symbolic link
1
contrib/arrow-cmake/build/cmake/config.h.in
Symbolic link
@ -0,0 +1 @@
|
|||||||
|
../../../thrift/build/cmake/config.h.in
|
||||||
17
contrib/arrow-cmake/cpp/src/parquet/parquet_constants.cpp
Normal file
17
contrib/arrow-cmake/cpp/src/parquet/parquet_constants.cpp
Normal file
@ -0,0 +1,17 @@
|
|||||||
|
/**
|
||||||
|
* Autogenerated by Thrift Compiler (0.11.0)
|
||||||
|
*
|
||||||
|
* DO NOT EDIT UNLESS YOU ARE SURE THAT YOU KNOW WHAT YOU ARE DOING
|
||||||
|
* @generated
|
||||||
|
*/
|
||||||
|
#include "parquet_constants.h"
|
||||||
|
|
||||||
|
namespace parquet { namespace format {
|
||||||
|
|
||||||
|
const parquetConstants g_parquet_constants;
|
||||||
|
|
||||||
|
parquetConstants::parquetConstants() {
|
||||||
|
}
|
||||||
|
|
||||||
|
}} // namespace
|
||||||
|
|
||||||
24
contrib/arrow-cmake/cpp/src/parquet/parquet_constants.h
Normal file
24
contrib/arrow-cmake/cpp/src/parquet/parquet_constants.h
Normal file
@ -0,0 +1,24 @@
|
|||||||
|
/**
|
||||||
|
* Autogenerated by Thrift Compiler (0.11.0)
|
||||||
|
*
|
||||||
|
* DO NOT EDIT UNLESS YOU ARE SURE THAT YOU KNOW WHAT YOU ARE DOING
|
||||||
|
* @generated
|
||||||
|
*/
|
||||||
|
#ifndef parquet_CONSTANTS_H
|
||||||
|
#define parquet_CONSTANTS_H
|
||||||
|
|
||||||
|
#include "parquet_types.h"
|
||||||
|
|
||||||
|
namespace parquet { namespace format {
|
||||||
|
|
||||||
|
class parquetConstants {
|
||||||
|
public:
|
||||||
|
parquetConstants();
|
||||||
|
|
||||||
|
};
|
||||||
|
|
||||||
|
extern const parquetConstants g_parquet_constants;
|
||||||
|
|
||||||
|
}} // namespace
|
||||||
|
|
||||||
|
#endif
|
||||||
6501
contrib/arrow-cmake/cpp/src/parquet/parquet_types.cpp
Normal file
6501
contrib/arrow-cmake/cpp/src/parquet/parquet_types.cpp
Normal file
File diff suppressed because it is too large
Load Diff
2523
contrib/arrow-cmake/cpp/src/parquet/parquet_types.h
Normal file
2523
contrib/arrow-cmake/cpp/src/parquet/parquet_types.h
Normal file
File diff suppressed because it is too large
Load Diff
24
contrib/arrow-cmake/cpp/src/parquet/parquet_version.h
Normal file
24
contrib/arrow-cmake/cpp/src/parquet/parquet_version.h
Normal file
@ -0,0 +1,24 @@
|
|||||||
|
// Licensed to the Apache Software Foundation (ASF) under one
|
||||||
|
// or more contributor license agreements. See the NOTICE file
|
||||||
|
// distributed with this work for additional information
|
||||||
|
// regarding copyright ownership. The ASF licenses this file
|
||||||
|
// to you under the Apache License, Version 2.0 (the
|
||||||
|
// "License"); you may not use this file except in compliance
|
||||||
|
// with the License. You may obtain a copy of the License at
|
||||||
|
//
|
||||||
|
// http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
//
|
||||||
|
// Unless required by applicable law or agreed to in writing,
|
||||||
|
// software distributed under the License is distributed on an
|
||||||
|
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||||
|
// KIND, either express or implied. See the License for the
|
||||||
|
// specific language governing permissions and limitations
|
||||||
|
// under the License.
|
||||||
|
|
||||||
|
#ifndef PARQUET_VERSION_H
|
||||||
|
#define PARQUET_VERSION_H
|
||||||
|
|
||||||
|
// define the parquet created by version
|
||||||
|
#define CREATED_BY_VERSION "parquet-cpp version 1.5.1-SNAPSHOT"
|
||||||
|
|
||||||
|
#endif // PARQUET_VERSION_H
|
||||||
11
contrib/arrow-cmake/cpp/src/thrift/stdcxx.h
Normal file
11
contrib/arrow-cmake/cpp/src/thrift/stdcxx.h
Normal file
@ -0,0 +1,11 @@
|
|||||||
|
/*
|
||||||
|
|
||||||
|
Temporary hack caused by 17355425 - THRIFT-4735: Remove Qt4 build support
|
||||||
|
|
||||||
|
Fixes
|
||||||
|
../contrib/arrow-cmake/cpp/src/parquet/parquet_types.h:18:10: fatal error: thrift/stdcxx.h: No such file or directory
|
||||||
|
#include <thrift/stdcxx.h>
|
||||||
|
|
||||||
|
Delete me.
|
||||||
|
|
||||||
|
*/
|
||||||
@ -39,5 +39,20 @@ add_library(base64 ${LINK_MODE}
|
|||||||
${LIBRARY_DIR}/lib/codecs.h
|
${LIBRARY_DIR}/lib/codecs.h
|
||||||
${CMAKE_CURRENT_BINARY_DIR}/config.h)
|
${CMAKE_CURRENT_BINARY_DIR}/config.h)
|
||||||
|
|
||||||
target_compile_options(base64 PRIVATE ${base64_SSSE3_opt} ${base64_SSE41_opt} ${base64_SSE42_opt} ${base64_AVX_opt} ${base64_AVX2_opt})
|
if(HAVE_AVX)
|
||||||
|
set_source_files_properties(${LIBRARY_DIR}/lib/arch/avx/codec.c PROPERTIES COMPILE_FLAGS -mavx)
|
||||||
|
endif()
|
||||||
|
if(HAVE_AVX2)
|
||||||
|
set_source_files_properties(${LIBRARY_DIR}/lib/arch/avx2/codec.c PROPERTIES COMPILE_FLAGS -mavx2)
|
||||||
|
endif()
|
||||||
|
if(HAVE_SSE41)
|
||||||
|
set_source_files_properties(${LIBRARY_DIR}/lib/arch/sse41/codec.c PROPERTIES COMPILE_FLAGS -msse4.1)
|
||||||
|
endif()
|
||||||
|
if(HAVE_SSE42)
|
||||||
|
set_source_files_properties(${LIBRARY_DIR}/lib/arch/sse42/codec.c PROPERTIES COMPILE_FLAGS -msse4.2)
|
||||||
|
endif()
|
||||||
|
if(HAVE_SSSE3)
|
||||||
|
set_source_files_properties(${LIBRARY_DIR}/lib/arch/ssse3/codec.c PROPERTIES COMPILE_FLAGS -mssse3)
|
||||||
|
endif()
|
||||||
|
|
||||||
target_include_directories(base64 PRIVATE ${LIBRARY_DIR}/include ${CMAKE_CURRENT_BINARY_DIR})
|
target_include_directories(base64 PRIVATE ${LIBRARY_DIR}/include ${CMAKE_CURRENT_BINARY_DIR})
|
||||||
|
|||||||
2
contrib/boost
vendored
2
contrib/boost
vendored
@ -1 +1 @@
|
|||||||
Subproject commit 6883b40449f378019aec792f9983ce3afc7ff16e
|
Subproject commit 6a96e8b59f76148eb8ad54a9d15259f8ce84c606
|
||||||
@ -10,49 +10,30 @@
|
|||||||
# Important boost patch: 094c18b
|
# Important boost patch: 094c18b
|
||||||
#
|
#
|
||||||
|
|
||||||
|
include(${ClickHouse_SOURCE_DIR}/cmake/dbms_glob_sources.cmake)
|
||||||
|
|
||||||
set(LIBRARY_DIR ${ClickHouse_SOURCE_DIR}/contrib/boost)
|
set(LIBRARY_DIR ${ClickHouse_SOURCE_DIR}/contrib/boost)
|
||||||
|
|
||||||
if(NOT MSVC)
|
if(NOT MSVC)
|
||||||
add_definitions(-Wno-unused-variable -Wno-deprecated-declarations)
|
add_definitions(-Wno-unused-variable -Wno-deprecated-declarations)
|
||||||
endif()
|
endif()
|
||||||
|
|
||||||
add_library(boost_program_options_internal ${LINK_MODE}
|
macro(add_boost_lib lib_name)
|
||||||
${LIBRARY_DIR}/libs/program_options/src/cmdline.cpp
|
add_headers_and_sources(boost_${lib_name} ${LIBRARY_DIR}/libs/${lib_name}/src)
|
||||||
${LIBRARY_DIR}/libs/program_options/src/config_file.cpp
|
add_library(boost_${lib_name}_internal ${LINK_MODE} ${boost_${lib_name}_sources})
|
||||||
${LIBRARY_DIR}/libs/program_options/src/convert.cpp
|
target_include_directories(boost_${lib_name}_internal SYSTEM BEFORE PUBLIC ${Boost_INCLUDE_DIRS})
|
||||||
${LIBRARY_DIR}/libs/program_options/src/options_description.cpp
|
target_compile_definitions(boost_${lib_name}_internal PUBLIC BOOST_SYSTEM_NO_DEPRECATED)
|
||||||
${LIBRARY_DIR}/libs/program_options/src/parsers.cpp
|
endmacro()
|
||||||
${LIBRARY_DIR}/libs/program_options/src/positional_options.cpp
|
|
||||||
${LIBRARY_DIR}/libs/program_options/src/split.cpp
|
|
||||||
${LIBRARY_DIR}/libs/program_options/src/utf8_codecvt_facet.cpp
|
|
||||||
${LIBRARY_DIR}/libs/program_options/src/value_semantic.cpp
|
|
||||||
${LIBRARY_DIR}/libs/program_options/src/variables_map.cpp
|
|
||||||
${LIBRARY_DIR}/libs/program_options/src/winmain.cpp)
|
|
||||||
|
|
||||||
add_library(boost_filesystem_internal ${LINK_MODE}
|
add_boost_lib(system)
|
||||||
${LIBRARY_DIR}/libs/filesystem/src/codecvt_error_category.cpp
|
|
||||||
${LIBRARY_DIR}/libs/filesystem/src/operations.cpp
|
|
||||||
${LIBRARY_DIR}/libs/filesystem/src/path.cpp
|
|
||||||
${LIBRARY_DIR}/libs/filesystem/src/path_traits.cpp
|
|
||||||
${LIBRARY_DIR}/libs/filesystem/src/portability.cpp
|
|
||||||
${LIBRARY_DIR}/libs/filesystem/src/unique_path.cpp
|
|
||||||
${LIBRARY_DIR}/libs/filesystem/src/utf8_codecvt_facet.cpp
|
|
||||||
${LIBRARY_DIR}/libs/filesystem/src/windows_file_codecvt.cpp)
|
|
||||||
|
|
||||||
add_library(boost_system_internal ${LINK_MODE}
|
add_boost_lib(program_options)
|
||||||
${LIBRARY_DIR}/libs/system/src/error_code.cpp)
|
|
||||||
|
|
||||||
add_library(boost_random_internal ${LINK_MODE}
|
add_boost_lib(filesystem)
|
||||||
${LIBRARY_DIR}/libs/random/src/random_device.cpp)
|
target_link_libraries(boost_filesystem_internal PRIVATE boost_system_internal)
|
||||||
|
|
||||||
target_link_libraries (boost_filesystem_internal PUBLIC boost_system_internal)
|
#add_boost_lib(random)
|
||||||
|
|
||||||
target_include_directories (boost_program_options_internal SYSTEM BEFORE PUBLIC ${Boost_INCLUDE_DIRS})
|
if (USE_INTERNAL_PARQUET_LIBRARY)
|
||||||
target_include_directories (boost_filesystem_internal SYSTEM BEFORE PUBLIC ${Boost_INCLUDE_DIRS})
|
add_boost_lib(regex)
|
||||||
target_include_directories (boost_system_internal SYSTEM BEFORE PUBLIC ${Boost_INCLUDE_DIRS})
|
endif()
|
||||||
target_include_directories (boost_random_internal SYSTEM BEFORE PUBLIC ${Boost_INCLUDE_DIRS})
|
|
||||||
|
|
||||||
target_compile_definitions (boost_program_options_internal PUBLIC BOOST_SYSTEM_NO_DEPRECATED)
|
|
||||||
target_compile_definitions (boost_filesystem_internal PUBLIC BOOST_SYSTEM_NO_DEPRECATED)
|
|
||||||
target_compile_definitions (boost_system_internal PUBLIC BOOST_SYSTEM_NO_DEPRECATED)
|
|
||||||
target_compile_definitions (boost_random_internal PUBLIC BOOST_SYSTEM_NO_DEPRECATED)
|
|
||||||
|
|||||||
1
contrib/brotli
vendored
Submodule
1
contrib/brotli
vendored
Submodule
@ -0,0 +1 @@
|
|||||||
|
Subproject commit 5805f99a533a8f8118699c0100d8c102f3605f65
|
||||||
33
contrib/brotli-cmake/CMakeLists.txt
Normal file
33
contrib/brotli-cmake/CMakeLists.txt
Normal file
@ -0,0 +1,33 @@
|
|||||||
|
set(BROTLI_SOURCE_DIR ${CMAKE_SOURCE_DIR}/contrib/brotli/c)
|
||||||
|
set(BROTLI_BINARY_DIR ${CMAKE_BINARY_DIR}/contrib/brotli/c)
|
||||||
|
|
||||||
|
set(SRCS
|
||||||
|
${BROTLI_SOURCE_DIR}/dec/bit_reader.c
|
||||||
|
${BROTLI_SOURCE_DIR}/dec/state.c
|
||||||
|
${BROTLI_SOURCE_DIR}/dec/huffman.c
|
||||||
|
${BROTLI_SOURCE_DIR}/dec/decode.c
|
||||||
|
${BROTLI_SOURCE_DIR}/enc/encode.c
|
||||||
|
${BROTLI_SOURCE_DIR}/enc/dictionary_hash.c
|
||||||
|
${BROTLI_SOURCE_DIR}/enc/cluster.c
|
||||||
|
${BROTLI_SOURCE_DIR}/enc/entropy_encode.c
|
||||||
|
${BROTLI_SOURCE_DIR}/enc/literal_cost.c
|
||||||
|
${BROTLI_SOURCE_DIR}/enc/compress_fragment_two_pass.c
|
||||||
|
${BROTLI_SOURCE_DIR}/enc/static_dict.c
|
||||||
|
${BROTLI_SOURCE_DIR}/enc/compress_fragment.c
|
||||||
|
${BROTLI_SOURCE_DIR}/enc/block_splitter.c
|
||||||
|
${BROTLI_SOURCE_DIR}/enc/backward_references_hq.c
|
||||||
|
${BROTLI_SOURCE_DIR}/enc/histogram.c
|
||||||
|
${BROTLI_SOURCE_DIR}/enc/brotli_bit_stream.c
|
||||||
|
${BROTLI_SOURCE_DIR}/enc/utf8_util.c
|
||||||
|
${BROTLI_SOURCE_DIR}/enc/encoder_dict.c
|
||||||
|
${BROTLI_SOURCE_DIR}/enc/backward_references.c
|
||||||
|
${BROTLI_SOURCE_DIR}/enc/bit_cost.c
|
||||||
|
${BROTLI_SOURCE_DIR}/enc/metablock.c
|
||||||
|
${BROTLI_SOURCE_DIR}/enc/memory.c
|
||||||
|
${BROTLI_SOURCE_DIR}/common/dictionary.c
|
||||||
|
${BROTLI_SOURCE_DIR}/common/transform.c
|
||||||
|
)
|
||||||
|
|
||||||
|
add_library(brotli ${LINK_MODE} ${SRCS})
|
||||||
|
|
||||||
|
target_include_directories(brotli PUBLIC ${BROTLI_SOURCE_DIR}/include)
|
||||||
1
contrib/cppkafka
vendored
Submodule
1
contrib/cppkafka
vendored
Submodule
@ -0,0 +1 @@
|
|||||||
|
Subproject commit 9b184d881c15cc50784b28688c7c99d3d764db24
|
||||||
31
contrib/cppkafka-cmake/CMakeLists.txt
Normal file
31
contrib/cppkafka-cmake/CMakeLists.txt
Normal file
@ -0,0 +1,31 @@
|
|||||||
|
set(CPPKAFKA_DIR ${CMAKE_SOURCE_DIR}/contrib/cppkafka)
|
||||||
|
|
||||||
|
set(SRCS
|
||||||
|
${CPPKAFKA_DIR}/src/configuration.cpp
|
||||||
|
${CPPKAFKA_DIR}/src/topic_configuration.cpp
|
||||||
|
${CPPKAFKA_DIR}/src/configuration_option.cpp
|
||||||
|
${CPPKAFKA_DIR}/src/exceptions.cpp
|
||||||
|
${CPPKAFKA_DIR}/src/topic.cpp
|
||||||
|
${CPPKAFKA_DIR}/src/buffer.cpp
|
||||||
|
${CPPKAFKA_DIR}/src/queue.cpp
|
||||||
|
${CPPKAFKA_DIR}/src/message.cpp
|
||||||
|
${CPPKAFKA_DIR}/src/message_timestamp.cpp
|
||||||
|
${CPPKAFKA_DIR}/src/message_internal.cpp
|
||||||
|
${CPPKAFKA_DIR}/src/topic_partition.cpp
|
||||||
|
${CPPKAFKA_DIR}/src/topic_partition_list.cpp
|
||||||
|
${CPPKAFKA_DIR}/src/metadata.cpp
|
||||||
|
${CPPKAFKA_DIR}/src/group_information.cpp
|
||||||
|
${CPPKAFKA_DIR}/src/error.cpp
|
||||||
|
${CPPKAFKA_DIR}/src/event.cpp
|
||||||
|
|
||||||
|
${CPPKAFKA_DIR}/src/kafka_handle_base.cpp
|
||||||
|
${CPPKAFKA_DIR}/src/producer.cpp
|
||||||
|
${CPPKAFKA_DIR}/src/consumer.cpp
|
||||||
|
)
|
||||||
|
|
||||||
|
add_library(cppkafka ${LINK_MODE} ${SRCS})
|
||||||
|
|
||||||
|
target_link_libraries(cppkafka PRIVATE ${RDKAFKA_LIBRARY})
|
||||||
|
target_include_directories(cppkafka PRIVATE ${CPPKAFKA_DIR}/include/cppkafka)
|
||||||
|
target_include_directories(cppkafka PRIVATE ${Boost_INCLUDE_DIRS})
|
||||||
|
target_include_directories(cppkafka SYSTEM PUBLIC ${CPPKAFKA_DIR}/include)
|
||||||
6
contrib/croaring/CMakeLists.txt
Normal file
6
contrib/croaring/CMakeLists.txt
Normal file
@ -0,0 +1,6 @@
|
|||||||
|
add_library(roaring
|
||||||
|
roaring.c
|
||||||
|
roaring.h
|
||||||
|
roaring.hh)
|
||||||
|
|
||||||
|
target_include_directories (roaring PUBLIC ${CMAKE_CURRENT_SOURCE_DIR})
|
||||||
202
contrib/croaring/LICENSE
Normal file
202
contrib/croaring/LICENSE
Normal file
@ -0,0 +1,202 @@
|
|||||||
|
Apache License
|
||||||
|
Version 2.0, January 2004
|
||||||
|
http://www.apache.org/licenses/
|
||||||
|
|
||||||
|
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||||
|
|
||||||
|
1. Definitions.
|
||||||
|
|
||||||
|
"License" shall mean the terms and conditions for use, reproduction,
|
||||||
|
and distribution as defined by Sections 1 through 9 of this document.
|
||||||
|
|
||||||
|
"Licensor" shall mean the copyright owner or entity authorized by
|
||||||
|
the copyright owner that is granting the License.
|
||||||
|
|
||||||
|
"Legal Entity" shall mean the union of the acting entity and all
|
||||||
|
other entities that control, are controlled by, or are under common
|
||||||
|
control with that entity. For the purposes of this definition,
|
||||||
|
"control" means (i) the power, direct or indirect, to cause the
|
||||||
|
direction or management of such entity, whether by contract or
|
||||||
|
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||||
|
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||||
|
|
||||||
|
"You" (or "Your") shall mean an individual or Legal Entity
|
||||||
|
exercising permissions granted by this License.
|
||||||
|
|
||||||
|
"Source" form shall mean the preferred form for making modifications,
|
||||||
|
including but not limited to software source code, documentation
|
||||||
|
source, and configuration files.
|
||||||
|
|
||||||
|
"Object" form shall mean any form resulting from mechanical
|
||||||
|
transformation or translation of a Source form, including but
|
||||||
|
not limited to compiled object code, generated documentation,
|
||||||
|
and conversions to other media types.
|
||||||
|
|
||||||
|
"Work" shall mean the work of authorship, whether in Source or
|
||||||
|
Object form, made available under the License, as indicated by a
|
||||||
|
copyright notice that is included in or attached to the work
|
||||||
|
(an example is provided in the Appendix below).
|
||||||
|
|
||||||
|
"Derivative Works" shall mean any work, whether in Source or Object
|
||||||
|
form, that is based on (or derived from) the Work and for which the
|
||||||
|
editorial revisions, annotations, elaborations, or other modifications
|
||||||
|
represent, as a whole, an original work of authorship. For the purposes
|
||||||
|
of this License, Derivative Works shall not include works that remain
|
||||||
|
separable from, or merely link (or bind by name) to the interfaces of,
|
||||||
|
the Work and Derivative Works thereof.
|
||||||
|
|
||||||
|
"Contribution" shall mean any work of authorship, including
|
||||||
|
the original version of the Work and any modifications or additions
|
||||||
|
to that Work or Derivative Works thereof, that is intentionally
|
||||||
|
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||||
|
or by an individual or Legal Entity authorized to submit on behalf of
|
||||||
|
the copyright owner. For the purposes of this definition, "submitted"
|
||||||
|
means any form of electronic, verbal, or written communication sent
|
||||||
|
to the Licensor or its representatives, including but not limited to
|
||||||
|
communication on electronic mailing lists, source code control systems,
|
||||||
|
and issue tracking systems that are managed by, or on behalf of, the
|
||||||
|
Licensor for the purpose of discussing and improving the Work, but
|
||||||
|
excluding communication that is conspicuously marked or otherwise
|
||||||
|
designated in writing by the copyright owner as "Not a Contribution."
|
||||||
|
|
||||||
|
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||||
|
on behalf of whom a Contribution has been received by Licensor and
|
||||||
|
subsequently incorporated within the Work.
|
||||||
|
|
||||||
|
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||||
|
this License, each Contributor hereby grants to You a perpetual,
|
||||||
|
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||||
|
copyright license to reproduce, prepare Derivative Works of,
|
||||||
|
publicly display, publicly perform, sublicense, and distribute the
|
||||||
|
Work and such Derivative Works in Source or Object form.
|
||||||
|
|
||||||
|
3. Grant of Patent License. Subject to the terms and conditions of
|
||||||
|
this License, each Contributor hereby grants to You a perpetual,
|
||||||
|
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||||
|
(except as stated in this section) patent license to make, have made,
|
||||||
|
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||||
|
where such license applies only to those patent claims licensable
|
||||||
|
by such Contributor that are necessarily infringed by their
|
||||||
|
Contribution(s) alone or by combination of their Contribution(s)
|
||||||
|
with the Work to which such Contribution(s) was submitted. If You
|
||||||
|
institute patent litigation against any entity (including a
|
||||||
|
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||||
|
or a Contribution incorporated within the Work constitutes direct
|
||||||
|
or contributory patent infringement, then any patent licenses
|
||||||
|
granted to You under this License for that Work shall terminate
|
||||||
|
as of the date such litigation is filed.
|
||||||
|
|
||||||
|
4. Redistribution. You may reproduce and distribute copies of the
|
||||||
|
Work or Derivative Works thereof in any medium, with or without
|
||||||
|
modifications, and in Source or Object form, provided that You
|
||||||
|
meet the following conditions:
|
||||||
|
|
||||||
|
(a) You must give any other recipients of the Work or
|
||||||
|
Derivative Works a copy of this License; and
|
||||||
|
|
||||||
|
(b) You must cause any modified files to carry prominent notices
|
||||||
|
stating that You changed the files; and
|
||||||
|
|
||||||
|
(c) You must retain, in the Source form of any Derivative Works
|
||||||
|
that You distribute, all copyright, patent, trademark, and
|
||||||
|
attribution notices from the Source form of the Work,
|
||||||
|
excluding those notices that do not pertain to any part of
|
||||||
|
the Derivative Works; and
|
||||||
|
|
||||||
|
(d) If the Work includes a "NOTICE" text file as part of its
|
||||||
|
distribution, then any Derivative Works that You distribute must
|
||||||
|
include a readable copy of the attribution notices contained
|
||||||
|
within such NOTICE file, excluding those notices that do not
|
||||||
|
pertain to any part of the Derivative Works, in at least one
|
||||||
|
of the following places: within a NOTICE text file distributed
|
||||||
|
as part of the Derivative Works; within the Source form or
|
||||||
|
documentation, if provided along with the Derivative Works; or,
|
||||||
|
within a display generated by the Derivative Works, if and
|
||||||
|
wherever such third-party notices normally appear. The contents
|
||||||
|
of the NOTICE file are for informational purposes only and
|
||||||
|
do not modify the License. You may add Your own attribution
|
||||||
|
notices within Derivative Works that You distribute, alongside
|
||||||
|
or as an addendum to the NOTICE text from the Work, provided
|
||||||
|
that such additional attribution notices cannot be construed
|
||||||
|
as modifying the License.
|
||||||
|
|
||||||
|
You may add Your own copyright statement to Your modifications and
|
||||||
|
may provide additional or different license terms and conditions
|
||||||
|
for use, reproduction, or distribution of Your modifications, or
|
||||||
|
for any such Derivative Works as a whole, provided Your use,
|
||||||
|
reproduction, and distribution of the Work otherwise complies with
|
||||||
|
the conditions stated in this License.
|
||||||
|
|
||||||
|
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||||
|
any Contribution intentionally submitted for inclusion in the Work
|
||||||
|
by You to the Licensor shall be under the terms and conditions of
|
||||||
|
this License, without any additional terms or conditions.
|
||||||
|
Notwithstanding the above, nothing herein shall supersede or modify
|
||||||
|
the terms of any separate license agreement you may have executed
|
||||||
|
with Licensor regarding such Contributions.
|
||||||
|
|
||||||
|
6. Trademarks. This License does not grant permission to use the trade
|
||||||
|
names, trademarks, service marks, or product names of the Licensor,
|
||||||
|
except as required for reasonable and customary use in describing the
|
||||||
|
origin of the Work and reproducing the content of the NOTICE file.
|
||||||
|
|
||||||
|
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||||
|
agreed to in writing, Licensor provides the Work (and each
|
||||||
|
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||||
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||||
|
implied, including, without limitation, any warranties or conditions
|
||||||
|
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||||
|
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||||
|
appropriateness of using or redistributing the Work and assume any
|
||||||
|
risks associated with Your exercise of permissions under this License.
|
||||||
|
|
||||||
|
8. Limitation of Liability. In no event and under no legal theory,
|
||||||
|
whether in tort (including negligence), contract, or otherwise,
|
||||||
|
unless required by applicable law (such as deliberate and grossly
|
||||||
|
negligent acts) or agreed to in writing, shall any Contributor be
|
||||||
|
liable to You for damages, including any direct, indirect, special,
|
||||||
|
incidental, or consequential damages of any character arising as a
|
||||||
|
result of this License or out of the use or inability to use the
|
||||||
|
Work (including but not limited to damages for loss of goodwill,
|
||||||
|
work stoppage, computer failure or malfunction, or any and all
|
||||||
|
other commercial damages or losses), even if such Contributor
|
||||||
|
has been advised of the possibility of such damages.
|
||||||
|
|
||||||
|
9. Accepting Warranty or Additional Liability. While redistributing
|
||||||
|
the Work or Derivative Works thereof, You may choose to offer,
|
||||||
|
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||||
|
or other liability obligations and/or rights consistent with this
|
||||||
|
License. However, in accepting such obligations, You may act only
|
||||||
|
on Your own behalf and on Your sole responsibility, not on behalf
|
||||||
|
of any other Contributor, and only if You agree to indemnify,
|
||||||
|
defend, and hold each Contributor harmless for any liability
|
||||||
|
incurred by, or claims asserted against, such Contributor by reason
|
||||||
|
of your accepting any such warranty or additional liability.
|
||||||
|
|
||||||
|
END OF TERMS AND CONDITIONS
|
||||||
|
|
||||||
|
APPENDIX: How to apply the Apache License to your work.
|
||||||
|
|
||||||
|
To apply the Apache License to your work, attach the following
|
||||||
|
boilerplate notice, with the fields enclosed by brackets "{}"
|
||||||
|
replaced with your own identifying information. (Don't include
|
||||||
|
the brackets!) The text should be enclosed in the appropriate
|
||||||
|
comment syntax for the file format. We also recommend that a
|
||||||
|
file or class name and description of purpose be included on the
|
||||||
|
same "printed page" as the copyright notice for easier
|
||||||
|
identification within third-party archives.
|
||||||
|
|
||||||
|
Copyright 2016 The CRoaring authors
|
||||||
|
|
||||||
|
Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
you may not use this file except in compliance with the License.
|
||||||
|
You may obtain a copy of the License at
|
||||||
|
|
||||||
|
http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
|
||||||
|
Unless required by applicable law or agreed to in writing, software
|
||||||
|
distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
See the License for the specific language governing permissions and
|
||||||
|
limitations under the License.
|
||||||
|
|
||||||
2
contrib/croaring/README.txt
Normal file
2
contrib/croaring/README.txt
Normal file
@ -0,0 +1,2 @@
|
|||||||
|
download from https://github.com/RoaringBitmap/CRoaring/archive/v0.2.57.tar.gz
|
||||||
|
and use ./amalgamation.sh generate
|
||||||
11093
contrib/croaring/roaring.c
Normal file
11093
contrib/croaring/roaring.c
Normal file
File diff suppressed because it is too large
Load Diff
7187
contrib/croaring/roaring.h
Normal file
7187
contrib/croaring/roaring.h
Normal file
File diff suppressed because it is too large
Load Diff
1732
contrib/croaring/roaring.hh
Normal file
1732
contrib/croaring/roaring.hh
Normal file
File diff suppressed because it is too large
Load Diff
@ -341,6 +341,13 @@ static inline __m128i libdivide_get_0000FFFF(void) {
|
|||||||
#pragma clang diagnostic pop
|
#pragma clang diagnostic pop
|
||||||
#endif
|
#endif
|
||||||
|
|
||||||
|
/// This is a bug in gcc-8, _MM_SHUFFLE was forgotten, though in trunk it is ok https://github.com/gcc-mirror/gcc/blob/master/gcc/config/rs6000/xmmintrin.h#L61
|
||||||
|
#if defined(__PPC__)
|
||||||
|
#ifndef _MM_SHUFFLE
|
||||||
|
#define _MM_SHUFFLE(w,x,y,z) (((w) << 6) | ((x) << 4) | ((y) << 2) | (z))
|
||||||
|
#endif
|
||||||
|
#endif
|
||||||
|
|
||||||
static inline __m128i libdivide_s64_signbits(__m128i v) {
|
static inline __m128i libdivide_s64_signbits(__m128i v) {
|
||||||
//we want to compute v >> 63, that is, _mm_srai_epi64(v, 63). But there is no 64 bit shift right arithmetic instruction in SSE2. So we have to fake it by first duplicating the high 32 bit values, and then using a 32 bit shift. Another option would be to use _mm_srli_epi64(v, 63) and then subtract that from 0, but that approach appears to be substantially slower for unknown reasons
|
//we want to compute v >> 63, that is, _mm_srai_epi64(v, 63). But there is no 64 bit shift right arithmetic instruction in SSE2. So we have to fake it by first duplicating the high 32 bit values, and then using a 32 bit shift. Another option would be to use _mm_srli_epi64(v, 63) and then subtract that from 0, but that approach appears to be substantially slower for unknown reasons
|
||||||
__m128i hiBitsDuped = _mm_shuffle_epi32(v, _MM_SHUFFLE(3, 3, 1, 1));
|
__m128i hiBitsDuped = _mm_shuffle_epi32(v, _MM_SHUFFLE(3, 3, 1, 1));
|
||||||
|
|||||||
@ -1,48 +0,0 @@
|
|||||||
# Check prereqs
|
|
||||||
FIND_PROGRAM(GCOV_PATH gcov)
|
|
||||||
FIND_PROGRAM(LCOV_PATH lcov)
|
|
||||||
FIND_PROGRAM(GENHTML_PATH genhtml)
|
|
||||||
|
|
||||||
IF(NOT GCOV_PATH)
|
|
||||||
MESSAGE(FATAL_ERROR "gcov not found! Aborting...")
|
|
||||||
ENDIF(NOT GCOV_PATH)
|
|
||||||
|
|
||||||
IF(NOT CMAKE_BUILD_TYPE STREQUAL Debug)
|
|
||||||
MESSAGE(WARNING "Code coverage results with an optimised (non-Debug) build may be misleading")
|
|
||||||
ENDIF(NOT CMAKE_BUILD_TYPE STREQUAL Debug)
|
|
||||||
|
|
||||||
#Setup compiler options
|
|
||||||
ADD_DEFINITIONS(-fprofile-arcs -ftest-coverage)
|
|
||||||
|
|
||||||
SET(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fprofile-arcs ")
|
|
||||||
SET(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -fprofile-arcs ")
|
|
||||||
|
|
||||||
IF(NOT LCOV_PATH)
|
|
||||||
MESSAGE(FATAL_ERROR "lcov not found! Aborting...")
|
|
||||||
ENDIF(NOT LCOV_PATH)
|
|
||||||
|
|
||||||
IF(NOT GENHTML_PATH)
|
|
||||||
MESSAGE(FATAL_ERROR "genhtml not found! Aborting...")
|
|
||||||
ENDIF(NOT GENHTML_PATH)
|
|
||||||
|
|
||||||
#Setup target
|
|
||||||
ADD_CUSTOM_TARGET(ShowCoverage
|
|
||||||
#Capturing lcov counters and generating report
|
|
||||||
COMMAND ${LCOV_PATH} --directory . --capture --output-file CodeCoverage.info
|
|
||||||
COMMAND ${LCOV_PATH} --remove CodeCoverage.info '${CMAKE_CURRENT_BINARY_DIR}/*' 'test/*' 'mock/*' '/usr/*' '/opt/*' '*ext/rhel5_x86_64*' '*ext/osx*' --output-file CodeCoverage.info.cleaned
|
|
||||||
COMMAND ${GENHTML_PATH} -o CodeCoverageReport CodeCoverage.info.cleaned
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
ADD_CUSTOM_TARGET(ShowAllCoverage
|
|
||||||
#Capturing lcov counters and generating report
|
|
||||||
COMMAND ${LCOV_PATH} -a CodeCoverage.info.cleaned -a CodeCoverage.info.cleaned_withoutHA -o AllCodeCoverage.info
|
|
||||||
COMMAND sed -e 's|/.*/src|${CMAKE_SOURCE_DIR}/src|' -ig AllCodeCoverage.info
|
|
||||||
COMMAND ${GENHTML_PATH} -o AllCodeCoverageReport AllCodeCoverage.info
|
|
||||||
)
|
|
||||||
|
|
||||||
ADD_CUSTOM_TARGET(ResetCoverage
|
|
||||||
#Cleanup lcov
|
|
||||||
COMMAND ${LCOV_PATH} --directory . --zerocounters
|
|
||||||
)
|
|
||||||
|
|
||||||
File diff suppressed because it is too large
Load Diff
@ -1,38 +1,15 @@
|
|||||||
OPTION(ENABLE_COVERAGE "enable code coverage" OFF)
|
|
||||||
OPTION(ENABLE_DEBUG "enable debug build" OFF)
|
|
||||||
OPTION(ENABLE_SSE "enable SSE4.2 buildin function" ON)
|
OPTION(ENABLE_SSE "enable SSE4.2 buildin function" ON)
|
||||||
OPTION(ENABLE_FRAME_POINTER "enable frame pointer on 64bit system with flag -fno-omit-frame-pointer, on 32bit system, it is always enabled" ON)
|
|
||||||
OPTION(ENABLE_LIBCPP "using libc++ instead of libstdc++, only valid for clang compiler" OFF)
|
|
||||||
OPTION(ENABLE_BOOST "using boost instead of native compiler c++0x support" OFF)
|
|
||||||
|
|
||||||
INCLUDE (CheckFunctionExists)
|
INCLUDE (CheckFunctionExists)
|
||||||
CHECK_FUNCTION_EXISTS(dladdr HAVE_DLADDR)
|
CHECK_FUNCTION_EXISTS(dladdr HAVE_DLADDR)
|
||||||
CHECK_FUNCTION_EXISTS(nanosleep HAVE_NANOSLEEP)
|
CHECK_FUNCTION_EXISTS(nanosleep HAVE_NANOSLEEP)
|
||||||
|
|
||||||
IF(ENABLE_DEBUG STREQUAL ON)
|
|
||||||
SET(CMAKE_BUILD_TYPE Debug CACHE
|
|
||||||
STRING "Choose the type of build, options are: None Debug Release RelWithDebInfo MinSizeRel." FORCE)
|
|
||||||
SET(CMAKE_CXX_FLAGS_DEBUG "-g -O0" CACHE STRING "compiler flags for debug" FORCE)
|
|
||||||
SET(CMAKE_C_FLAGS_DEBUG "-g -O0" CACHE STRING "compiler flags for debug" FORCE)
|
|
||||||
ELSE(ENABLE_DEBUG STREQUAL ON)
|
|
||||||
SET(CMAKE_BUILD_TYPE RelWithDebInfo CACHE
|
|
||||||
STRING "Choose the type of build, options are: None Debug Release RelWithDebInfo MinSizeRel." FORCE)
|
|
||||||
ENDIF(ENABLE_DEBUG STREQUAL ON)
|
|
||||||
|
|
||||||
SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fno-strict-aliasing")
|
SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fno-strict-aliasing")
|
||||||
SET(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fno-strict-aliasing")
|
SET(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fno-strict-aliasing")
|
||||||
|
|
||||||
IF(ENABLE_COVERAGE STREQUAL ON)
|
IF(ENABLE_SSE STREQUAL ON AND NOT ARCH_PPC64LE AND NOT ARCH_AARCH64 AND NOT ARCH_ARM)
|
||||||
INCLUDE(CodeCoverage)
|
|
||||||
ENDIF(ENABLE_COVERAGE STREQUAL ON)
|
|
||||||
|
|
||||||
IF(ENABLE_FRAME_POINTER STREQUAL ON)
|
|
||||||
SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fno-omit-frame-pointer")
|
|
||||||
ENDIF(ENABLE_FRAME_POINTER STREQUAL ON)
|
|
||||||
|
|
||||||
IF(ENABLE_SSE STREQUAL ON)
|
|
||||||
SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -msse4.2")
|
SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -msse4.2")
|
||||||
ENDIF(ENABLE_SSE STREQUAL ON)
|
ENDIF(ENABLE_SSE STREQUAL ON AND NOT ARCH_PPC64LE AND NOT ARCH_AARCH64 AND NOT ARCH_ARM)
|
||||||
|
|
||||||
IF(NOT TEST_HDFS_PREFIX)
|
IF(NOT TEST_HDFS_PREFIX)
|
||||||
SET(TEST_HDFS_PREFIX "./" CACHE STRING "default directory prefix used for test." FORCE)
|
SET(TEST_HDFS_PREFIX "./" CACHE STRING "default directory prefix used for test." FORCE)
|
||||||
@ -41,76 +18,7 @@ ENDIF(NOT TEST_HDFS_PREFIX)
|
|||||||
ADD_DEFINITIONS(-DTEST_HDFS_PREFIX="${TEST_HDFS_PREFIX}")
|
ADD_DEFINITIONS(-DTEST_HDFS_PREFIX="${TEST_HDFS_PREFIX}")
|
||||||
ADD_DEFINITIONS(-D__STDC_FORMAT_MACROS)
|
ADD_DEFINITIONS(-D__STDC_FORMAT_MACROS)
|
||||||
ADD_DEFINITIONS(-D_GNU_SOURCE)
|
ADD_DEFINITIONS(-D_GNU_SOURCE)
|
||||||
|
ADD_DEFINITIONS(-D_GLIBCXX_USE_NANOSLEEP)
|
||||||
IF(OS_MACOSX AND CMAKE_COMPILER_IS_GNUCXX)
|
|
||||||
SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wl,-bind_at_load")
|
|
||||||
ENDIF(OS_MACOSX AND CMAKE_COMPILER_IS_GNUCXX)
|
|
||||||
|
|
||||||
IF(OS_LINUX)
|
|
||||||
SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wl,--export-dynamic")
|
|
||||||
ENDIF(OS_LINUX)
|
|
||||||
|
|
||||||
SET(BOOST_ROOT ${CMAKE_PREFIX_PATH})
|
|
||||||
IF(ENABLE_BOOST STREQUAL ON)
|
|
||||||
MESSAGE(STATUS "using boost instead of native compiler c++0x support.")
|
|
||||||
FIND_PACKAGE(Boost 1.50 REQUIRED)
|
|
||||||
SET(NEED_BOOST true CACHE INTERNAL "boost is required")
|
|
||||||
ELSE(ENABLE_BOOST STREQUAL ON)
|
|
||||||
SET(NEED_BOOST false CACHE INTERNAL "boost is required")
|
|
||||||
ENDIF(ENABLE_BOOST STREQUAL ON)
|
|
||||||
|
|
||||||
IF(CMAKE_COMPILER_IS_GNUCXX)
|
|
||||||
IF(ENABLE_LIBCPP STREQUAL ON)
|
|
||||||
MESSAGE(FATAL_ERROR "Unsupport using GCC compiler with libc++")
|
|
||||||
ENDIF(ENABLE_LIBCPP STREQUAL ON)
|
|
||||||
|
|
||||||
IF((GCC_COMPILER_VERSION_MAJOR EQUAL 4) AND (GCC_COMPILER_VERSION_MINOR EQUAL 4) AND OS_MACOSX)
|
|
||||||
SET(NEED_GCCEH true CACHE INTERNAL "Explicitly link with gcc_eh")
|
|
||||||
MESSAGE(STATUS "link with -lgcc_eh for TLS")
|
|
||||||
ENDIF((GCC_COMPILER_VERSION_MAJOR EQUAL 4) AND (GCC_COMPILER_VERSION_MINOR EQUAL 4) AND OS_MACOSX)
|
|
||||||
|
|
||||||
IF((GCC_COMPILER_VERSION_MAJOR LESS 4) OR ((GCC_COMPILER_VERSION_MAJOR EQUAL 4) AND (GCC_COMPILER_VERSION_MINOR LESS 4)))
|
|
||||||
SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall")
|
|
||||||
IF(NOT ENABLE_BOOST STREQUAL ON)
|
|
||||||
MESSAGE(STATUS "gcc version is older than 4.6.0, boost is required.")
|
|
||||||
FIND_PACKAGE(Boost 1.50 REQUIRED)
|
|
||||||
SET(NEED_BOOST true CACHE INTERNAL "boost is required")
|
|
||||||
ENDIF(NOT ENABLE_BOOST STREQUAL ON)
|
|
||||||
ELSEIF((GCC_COMPILER_VERSION_MAJOR EQUAL 4) AND (GCC_COMPILER_VERSION_MINOR LESS 7))
|
|
||||||
IF(NOT ENABLE_BOOST STREQUAL ON)
|
|
||||||
MESSAGE(STATUS "gcc version is older than 4.6.0, boost is required.")
|
|
||||||
FIND_PACKAGE(Boost 1.50 REQUIRED)
|
|
||||||
SET(NEED_BOOST true CACHE INTERNAL "boost is required")
|
|
||||||
ENDIF(NOT ENABLE_BOOST STREQUAL ON)
|
|
||||||
MESSAGE(STATUS "adding c++0x support for gcc compiler")
|
|
||||||
SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++0x")
|
|
||||||
ELSE((GCC_COMPILER_VERSION_MAJOR LESS 4) OR ((GCC_COMPILER_VERSION_MAJOR EQUAL 4) AND (GCC_COMPILER_VERSION_MINOR LESS 4)))
|
|
||||||
MESSAGE(STATUS "adding c++0x support for gcc compiler")
|
|
||||||
SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++0x")
|
|
||||||
ENDIF((GCC_COMPILER_VERSION_MAJOR LESS 4) OR ((GCC_COMPILER_VERSION_MAJOR EQUAL 4) AND (GCC_COMPILER_VERSION_MINOR LESS 4)))
|
|
||||||
|
|
||||||
IF(NEED_BOOST)
|
|
||||||
IF((Boost_MAJOR_VERSION LESS 1) OR ((Boost_MAJOR_VERSION EQUAL 1) AND (Boost_MINOR_VERSION LESS 50)))
|
|
||||||
MESSAGE(FATAL_ERROR "boost 1.50+ is required")
|
|
||||||
ENDIF()
|
|
||||||
ELSE(NEED_BOOST)
|
|
||||||
IF(HAVE_NANOSLEEP)
|
|
||||||
SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -D_GLIBCXX_USE_NANOSLEEP")
|
|
||||||
ELSE(HAVE_NANOSLEEP)
|
|
||||||
MESSAGE(FATAL_ERROR "nanosleep() is required")
|
|
||||||
ENDIF(HAVE_NANOSLEEP)
|
|
||||||
ENDIF(NEED_BOOST)
|
|
||||||
SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall")
|
|
||||||
ELSEIF(CMAKE_COMPILER_IS_CLANG)
|
|
||||||
MESSAGE(STATUS "adding c++0x support for clang compiler")
|
|
||||||
SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++0x")
|
|
||||||
SET(CMAKE_XCODE_ATTRIBUTE_CLANG_CXX_LANGUAGE_STANDARD "c++0x")
|
|
||||||
IF(ENABLE_LIBCPP STREQUAL ON)
|
|
||||||
MESSAGE(STATUS "using libc++ instead of libstdc++")
|
|
||||||
SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -stdlib=libc++")
|
|
||||||
SET(CMAKE_XCODE_ATTRIBUTE_CLANG_CXX_LIBRARY "libc++")
|
|
||||||
ENDIF(ENABLE_LIBCPP STREQUAL ON)
|
|
||||||
ENDIF(CMAKE_COMPILER_IS_GNUCXX)
|
|
||||||
|
|
||||||
TRY_COMPILE(STRERROR_R_RETURN_INT
|
TRY_COMPILE(STRERROR_R_RETURN_INT
|
||||||
${CMAKE_CURRENT_BINARY_DIR}
|
${CMAKE_CURRENT_BINARY_DIR}
|
||||||
@ -138,32 +46,8 @@ TRY_COMPILE(HAVE_NESTED_EXCEPTION
|
|||||||
CMAKE_FLAGS "-DCMAKE_CXX_LINK_EXECUTABLE='echo not linking now...'"
|
CMAKE_FLAGS "-DCMAKE_CXX_LINK_EXECUTABLE='echo not linking now...'"
|
||||||
OUTPUT_VARIABLE OUTPUT)
|
OUTPUT_VARIABLE OUTPUT)
|
||||||
|
|
||||||
FILE(WRITE ${CMAKE_CURRENT_BINARY_DIR}/test.cpp "#include <boost/chrono.hpp>")
|
SET(HAVE_BOOST_CHRONO 0)
|
||||||
TRY_COMPILE(HAVE_BOOST_CHRONO
|
SET(HAVE_BOOST_ATOMIC 0)
|
||||||
${CMAKE_CURRENT_BINARY_DIR}
|
|
||||||
${CMAKE_CURRENT_BINARY_DIR}/test.cpp
|
|
||||||
CMAKE_FLAGS "-DCMAKE_CXX_LINK_EXECUTABLE='echo not linking now...'"
|
|
||||||
-DINCLUDE_DIRECTORIES=${Boost_INCLUDE_DIR}
|
|
||||||
OUTPUT_VARIABLE OUTPUT)
|
|
||||||
|
|
||||||
FILE(WRITE ${CMAKE_CURRENT_BINARY_DIR}/test.cpp "#include <chrono>")
|
SET(HAVE_STD_CHRONO 1)
|
||||||
TRY_COMPILE(HAVE_STD_CHRONO
|
SET(HAVE_STD_ATOMIC 1)
|
||||||
${CMAKE_CURRENT_BINARY_DIR}
|
|
||||||
${CMAKE_CURRENT_BINARY_DIR}/test.cpp
|
|
||||||
CMAKE_FLAGS "-DCMAKE_CXX_LINK_EXECUTABLE='echo not linking now...'"
|
|
||||||
OUTPUT_VARIABLE OUTPUT)
|
|
||||||
|
|
||||||
FILE(WRITE ${CMAKE_CURRENT_BINARY_DIR}/test.cpp "#include <boost/atomic.hpp>")
|
|
||||||
TRY_COMPILE(HAVE_BOOST_ATOMIC
|
|
||||||
${CMAKE_CURRENT_BINARY_DIR}
|
|
||||||
${CMAKE_CURRENT_BINARY_DIR}/test.cpp
|
|
||||||
CMAKE_FLAGS "-DCMAKE_CXX_LINK_EXECUTABLE='echo not linking now...'"
|
|
||||||
-DINCLUDE_DIRECTORIES=${Boost_INCLUDE_DIR}
|
|
||||||
OUTPUT_VARIABLE OUTPUT)
|
|
||||||
|
|
||||||
FILE(WRITE ${CMAKE_CURRENT_BINARY_DIR}/test.cpp "#include <atomic>")
|
|
||||||
TRY_COMPILE(HAVE_STD_ATOMIC
|
|
||||||
${CMAKE_CURRENT_BINARY_DIR}
|
|
||||||
${CMAKE_CURRENT_BINARY_DIR}/test.cpp
|
|
||||||
CMAKE_FLAGS "-DCMAKE_CXX_LINK_EXECUTABLE='echo not linking now...'"
|
|
||||||
OUTPUT_VARIABLE OUTPUT)
|
|
||||||
|
|||||||
@ -36,7 +36,9 @@ set(PROTO_FILES
|
|||||||
${HDFS3_SOURCE_DIR}/proto/datatransfer.proto
|
${HDFS3_SOURCE_DIR}/proto/datatransfer.proto
|
||||||
)
|
)
|
||||||
|
|
||||||
|
if(USE_PROTOBUF)
|
||||||
PROTOBUF_GENERATE_CPP(PROTO_SOURCES PROTO_HEADERS ${PROTO_FILES})
|
PROTOBUF_GENERATE_CPP(PROTO_SOURCES PROTO_HEADERS ${PROTO_FILES})
|
||||||
|
endif()
|
||||||
|
|
||||||
configure_file(${HDFS3_SOURCE_DIR}/platform.h.in ${CMAKE_CURRENT_BINARY_DIR}/platform.h)
|
configure_file(${HDFS3_SOURCE_DIR}/platform.h.in ${CMAKE_CURRENT_BINARY_DIR}/platform.h)
|
||||||
|
|
||||||
|
|||||||
@ -1,5 +1,5 @@
|
|||||||
if (HAVE_SSE42) # Not used. Pretty easy to port.
|
if (HAVE_SSE42) # Not used. Pretty easy to port.
|
||||||
set (SOURCES_SSE42_ONLY src/metrohash128crc.cpp)
|
set (SOURCES_SSE42_ONLY src/metrohash128crc.cpp src/metrohash128crc.h)
|
||||||
endif ()
|
endif ()
|
||||||
|
|
||||||
add_library(metrohash
|
add_library(metrohash
|
||||||
|
|||||||
@ -1,22 +1,201 @@
|
|||||||
The MIT License (MIT)
|
Apache License
|
||||||
|
Version 2.0, January 2004
|
||||||
|
http://www.apache.org/licenses/
|
||||||
|
|
||||||
Copyright (c) 2015 J. Andrew Rogers
|
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||||
|
|
||||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
1. Definitions.
|
||||||
of this software and associated documentation files (the "Software"), to deal
|
|
||||||
in the Software without restriction, including without limitation the rights
|
|
||||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
||||||
copies of the Software, and to permit persons to whom the Software is
|
|
||||||
furnished to do so, subject to the following conditions:
|
|
||||||
|
|
||||||
The above copyright notice and this permission notice shall be included in all
|
"License" shall mean the terms and conditions for use, reproduction,
|
||||||
copies or substantial portions of the Software.
|
and distribution as defined by Sections 1 through 9 of this document.
|
||||||
|
|
||||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
"Licensor" shall mean the copyright owner or entity authorized by
|
||||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
the copyright owner that is granting the License.
|
||||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
||||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
||||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
||||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
||||||
SOFTWARE.
|
|
||||||
|
|
||||||
|
"Legal Entity" shall mean the union of the acting entity and all
|
||||||
|
other entities that control, are controlled by, or are under common
|
||||||
|
control with that entity. For the purposes of this definition,
|
||||||
|
"control" means (i) the power, direct or indirect, to cause the
|
||||||
|
direction or management of such entity, whether by contract or
|
||||||
|
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||||
|
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||||
|
|
||||||
|
"You" (or "Your") shall mean an individual or Legal Entity
|
||||||
|
exercising permissions granted by this License.
|
||||||
|
|
||||||
|
"Source" form shall mean the preferred form for making modifications,
|
||||||
|
including but not limited to software source code, documentation
|
||||||
|
source, and configuration files.
|
||||||
|
|
||||||
|
"Object" form shall mean any form resulting from mechanical
|
||||||
|
transformation or translation of a Source form, including but
|
||||||
|
not limited to compiled object code, generated documentation,
|
||||||
|
and conversions to other media types.
|
||||||
|
|
||||||
|
"Work" shall mean the work of authorship, whether in Source or
|
||||||
|
Object form, made available under the License, as indicated by a
|
||||||
|
copyright notice that is included in or attached to the work
|
||||||
|
(an example is provided in the Appendix below).
|
||||||
|
|
||||||
|
"Derivative Works" shall mean any work, whether in Source or Object
|
||||||
|
form, that is based on (or derived from) the Work and for which the
|
||||||
|
editorial revisions, annotations, elaborations, or other modifications
|
||||||
|
represent, as a whole, an original work of authorship. For the purposes
|
||||||
|
of this License, Derivative Works shall not include works that remain
|
||||||
|
separable from, or merely link (or bind by name) to the interfaces of,
|
||||||
|
the Work and Derivative Works thereof.
|
||||||
|
|
||||||
|
"Contribution" shall mean any work of authorship, including
|
||||||
|
the original version of the Work and any modifications or additions
|
||||||
|
to that Work or Derivative Works thereof, that is intentionally
|
||||||
|
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||||
|
or by an individual or Legal Entity authorized to submit on behalf of
|
||||||
|
the copyright owner. For the purposes of this definition, "submitted"
|
||||||
|
means any form of electronic, verbal, or written communication sent
|
||||||
|
to the Licensor or its representatives, including but not limited to
|
||||||
|
communication on electronic mailing lists, source code control systems,
|
||||||
|
and issue tracking systems that are managed by, or on behalf of, the
|
||||||
|
Licensor for the purpose of discussing and improving the Work, but
|
||||||
|
excluding communication that is conspicuously marked or otherwise
|
||||||
|
designated in writing by the copyright owner as "Not a Contribution."
|
||||||
|
|
||||||
|
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||||
|
on behalf of whom a Contribution has been received by Licensor and
|
||||||
|
subsequently incorporated within the Work.
|
||||||
|
|
||||||
|
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||||
|
this License, each Contributor hereby grants to You a perpetual,
|
||||||
|
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||||
|
copyright license to reproduce, prepare Derivative Works of,
|
||||||
|
publicly display, publicly perform, sublicense, and distribute the
|
||||||
|
Work and such Derivative Works in Source or Object form.
|
||||||
|
|
||||||
|
3. Grant of Patent License. Subject to the terms and conditions of
|
||||||
|
this License, each Contributor hereby grants to You a perpetual,
|
||||||
|
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||||
|
(except as stated in this section) patent license to make, have made,
|
||||||
|
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||||
|
where such license applies only to those patent claims licensable
|
||||||
|
by such Contributor that are necessarily infringed by their
|
||||||
|
Contribution(s) alone or by combination of their Contribution(s)
|
||||||
|
with the Work to which such Contribution(s) was submitted. If You
|
||||||
|
institute patent litigation against any entity (including a
|
||||||
|
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||||
|
or a Contribution incorporated within the Work constitutes direct
|
||||||
|
or contributory patent infringement, then any patent licenses
|
||||||
|
granted to You under this License for that Work shall terminate
|
||||||
|
as of the date such litigation is filed.
|
||||||
|
|
||||||
|
4. Redistribution. You may reproduce and distribute copies of the
|
||||||
|
Work or Derivative Works thereof in any medium, with or without
|
||||||
|
modifications, and in Source or Object form, provided that You
|
||||||
|
meet the following conditions:
|
||||||
|
|
||||||
|
(a) You must give any other recipients of the Work or
|
||||||
|
Derivative Works a copy of this License; and
|
||||||
|
|
||||||
|
(b) You must cause any modified files to carry prominent notices
|
||||||
|
stating that You changed the files; and
|
||||||
|
|
||||||
|
(c) You must retain, in the Source form of any Derivative Works
|
||||||
|
that You distribute, all copyright, patent, trademark, and
|
||||||
|
attribution notices from the Source form of the Work,
|
||||||
|
excluding those notices that do not pertain to any part of
|
||||||
|
the Derivative Works; and
|
||||||
|
|
||||||
|
(d) If the Work includes a "NOTICE" text file as part of its
|
||||||
|
distribution, then any Derivative Works that You distribute must
|
||||||
|
include a readable copy of the attribution notices contained
|
||||||
|
within such NOTICE file, excluding those notices that do not
|
||||||
|
pertain to any part of the Derivative Works, in at least one
|
||||||
|
of the following places: within a NOTICE text file distributed
|
||||||
|
as part of the Derivative Works; within the Source form or
|
||||||
|
documentation, if provided along with the Derivative Works; or,
|
||||||
|
within a display generated by the Derivative Works, if and
|
||||||
|
wherever such third-party notices normally appear. The contents
|
||||||
|
of the NOTICE file are for informational purposes only and
|
||||||
|
do not modify the License. You may add Your own attribution
|
||||||
|
notices within Derivative Works that You distribute, alongside
|
||||||
|
or as an addendum to the NOTICE text from the Work, provided
|
||||||
|
that such additional attribution notices cannot be construed
|
||||||
|
as modifying the License.
|
||||||
|
|
||||||
|
You may add Your own copyright statement to Your modifications and
|
||||||
|
may provide additional or different license terms and conditions
|
||||||
|
for use, reproduction, or distribution of Your modifications, or
|
||||||
|
for any such Derivative Works as a whole, provided Your use,
|
||||||
|
reproduction, and distribution of the Work otherwise complies with
|
||||||
|
the conditions stated in this License.
|
||||||
|
|
||||||
|
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||||
|
any Contribution intentionally submitted for inclusion in the Work
|
||||||
|
by You to the Licensor shall be under the terms and conditions of
|
||||||
|
this License, without any additional terms or conditions.
|
||||||
|
Notwithstanding the above, nothing herein shall supersede or modify
|
||||||
|
the terms of any separate license agreement you may have executed
|
||||||
|
with Licensor regarding such Contributions.
|
||||||
|
|
||||||
|
6. Trademarks. This License does not grant permission to use the trade
|
||||||
|
names, trademarks, service marks, or product names of the Licensor,
|
||||||
|
except as required for reasonable and customary use in describing the
|
||||||
|
origin of the Work and reproducing the content of the NOTICE file.
|
||||||
|
|
||||||
|
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||||
|
agreed to in writing, Licensor provides the Work (and each
|
||||||
|
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||||
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||||
|
implied, including, without limitation, any warranties or conditions
|
||||||
|
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||||
|
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||||
|
appropriateness of using or redistributing the Work and assume any
|
||||||
|
risks associated with Your exercise of permissions under this License.
|
||||||
|
|
||||||
|
8. Limitation of Liability. In no event and under no legal theory,
|
||||||
|
whether in tort (including negligence), contract, or otherwise,
|
||||||
|
unless required by applicable law (such as deliberate and grossly
|
||||||
|
negligent acts) or agreed to in writing, shall any Contributor be
|
||||||
|
liable to You for damages, including any direct, indirect, special,
|
||||||
|
incidental, or consequential damages of any character arising as a
|
||||||
|
result of this License or out of the use or inability to use the
|
||||||
|
Work (including but not limited to damages for loss of goodwill,
|
||||||
|
work stoppage, computer failure or malfunction, or any and all
|
||||||
|
other commercial damages or losses), even if such Contributor
|
||||||
|
has been advised of the possibility of such damages.
|
||||||
|
|
||||||
|
9. Accepting Warranty or Additional Liability. While redistributing
|
||||||
|
the Work or Derivative Works thereof, You may choose to offer,
|
||||||
|
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||||
|
or other liability obligations and/or rights consistent with this
|
||||||
|
License. However, in accepting such obligations, You may act only
|
||||||
|
on Your own behalf and on Your sole responsibility, not on behalf
|
||||||
|
of any other Contributor, and only if You agree to indemnify,
|
||||||
|
defend, and hold each Contributor harmless for any liability
|
||||||
|
incurred by, or claims asserted against, such Contributor by reason
|
||||||
|
of your accepting any such warranty or additional liability.
|
||||||
|
|
||||||
|
END OF TERMS AND CONDITIONS
|
||||||
|
|
||||||
|
APPENDIX: How to apply the Apache License to your work.
|
||||||
|
|
||||||
|
To apply the Apache License to your work, attach the following
|
||||||
|
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||||
|
replaced with your own identifying information. (Don't include
|
||||||
|
the brackets!) The text should be enclosed in the appropriate
|
||||||
|
comment syntax for the file format. We also recommend that a
|
||||||
|
file or class name and description of purpose be included on the
|
||||||
|
same "printed page" as the copyright notice for easier
|
||||||
|
identification within third-party archives.
|
||||||
|
|
||||||
|
Copyright [yyyy] [name of copyright owner]
|
||||||
|
|
||||||
|
Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
you may not use this file except in compliance with the License.
|
||||||
|
You may obtain a copy of the License at
|
||||||
|
|
||||||
|
http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
|
||||||
|
Unless required by applicable law or agreed to in writing, software
|
||||||
|
distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
See the License for the specific language governing permissions and
|
||||||
|
limitations under the License.
|
||||||
|
|||||||
@ -5,12 +5,44 @@ MetroHash is a set of state-of-the-art hash functions for *non-cryptographic* us
|
|||||||
* Fastest general-purpose functions for bulk hashing.
|
* Fastest general-purpose functions for bulk hashing.
|
||||||
* Fastest general-purpose functions for small, variable length keys.
|
* Fastest general-purpose functions for small, variable length keys.
|
||||||
* Robust statistical bias profile, similar to the MD5 cryptographic hash.
|
* Robust statistical bias profile, similar to the MD5 cryptographic hash.
|
||||||
|
* Hashes can be constructed incrementally (**new**)
|
||||||
* 64-bit, 128-bit, and 128-bit CRC variants currently available.
|
* 64-bit, 128-bit, and 128-bit CRC variants currently available.
|
||||||
* Optimized for modern x86-64 microarchitectures.
|
* Optimized for modern x86-64 microarchitectures.
|
||||||
* Elegant, compact, readable functions.
|
* Elegant, compact, readable functions.
|
||||||
|
|
||||||
You can read more about the design and history [here](http://www.jandrewrogers.com/2015/05/27/metrohash/).
|
You can read more about the design and history [here](http://www.jandrewrogers.com/2015/05/27/metrohash/).
|
||||||
|
|
||||||
|
## News
|
||||||
|
|
||||||
|
### 23 October 2018
|
||||||
|
|
||||||
|
The project has been re-licensed under Apache License v2.0. The purpose of this license change is consistency with the imminent release of MetroHash v2.0, which is also licensed under the Apache license.
|
||||||
|
|
||||||
|
### 27 July 2015
|
||||||
|
|
||||||
|
Two new 64-bit and 128-bit algorithms add the ability to construct hashes incrementally. In addition to supporting incremental construction, the algorithms are slightly superior to the prior versions.
|
||||||
|
|
||||||
|
A big change is that these new algorithms are implemented as C++ classes that support both incremental and stateless hashing. These classes also have a static method for verifying the implementation against the test vectors built into the classes. Implementations are now fully contained by their respective headers e.g. "metrohash128.h".
|
||||||
|
|
||||||
|
*Note: an incremental version of the 128-bit CRC version is on its way but is not included in this push.*
|
||||||
|
|
||||||
|
**Usage Example For Stateless Hashing**
|
||||||
|
|
||||||
|
`MetroHash128::Hash(key, key_length, hash_ptr, seed)`
|
||||||
|
|
||||||
|
**Usage Example For Incremental Hashing**
|
||||||
|
|
||||||
|
`MetroHash128 hasher;`
|
||||||
|
`hasher.Update(partial_key, partial_key_length);`
|
||||||
|
`...`
|
||||||
|
`hasher.Update(partial_key, partial_key_length);`
|
||||||
|
`hasher.Finalize(hash_ptr);`
|
||||||
|
|
||||||
|
An `Initialize(seed)` method allows the hasher objects to be reused.
|
||||||
|
|
||||||
|
|
||||||
|
### 27 May 2015
|
||||||
|
|
||||||
Six hash functions have been included in the initial release:
|
Six hash functions have been included in the initial release:
|
||||||
|
|
||||||
* 64-bit hash functions, "metrohash64_1" and "metrohash64_2"
|
* 64-bit hash functions, "metrohash64_1" and "metrohash64_2"
|
||||||
|
|||||||
@ -1,7 +1,4 @@
|
|||||||
origin: git@github.com:jandrewrogers/MetroHash.git
|
origin: https://github.com/jandrewrogers/MetroHash.git
|
||||||
commit d9dee18a54a8a6766e24c1950b814ac7ca9d1a89
|
commit 690a521d9beb2e1050cc8f273fdabc13b31bf8f6 tag: v1.1.3
|
||||||
Merge: 761e8a4 3d06b24
|
|
||||||
Author: J. Andrew Rogers <andrew@jarbox.org>
|
Author: J. Andrew Rogers <andrew@jarbox.org>
|
||||||
Date: Sat Jun 6 16:12:06 2015 -0700
|
Date: Tue Oct 23 09:49:53 2018 -0700
|
||||||
|
|
||||||
modified README
|
|
||||||
|
|||||||
@ -1,73 +1,24 @@
|
|||||||
// metrohash.h
|
// metrohash.h
|
||||||
//
|
//
|
||||||
// The MIT License (MIT)
|
// Copyright 2015-2018 J. Andrew Rogers
|
||||||
//
|
//
|
||||||
// Copyright (c) 2015 J. Andrew Rogers
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
// you may not use this file except in compliance with the License.
|
||||||
|
// You may obtain a copy of the License at
|
||||||
//
|
//
|
||||||
// Permission is hereby granted, free of charge, to any person obtaining a copy
|
// http://www.apache.org/licenses/LICENSE-2.0
|
||||||
// of this software and associated documentation files (the "Software"), to deal
|
|
||||||
// in the Software without restriction, including without limitation the rights
|
|
||||||
// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
||||||
// copies of the Software, and to permit persons to whom the Software is
|
|
||||||
// furnished to do so, subject to the following conditions:
|
|
||||||
//
|
|
||||||
// The above copyright notice and this permission notice shall be included in all
|
|
||||||
// copies or substantial portions of the Software.
|
|
||||||
//
|
|
||||||
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
||||||
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
||||||
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
||||||
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
||||||
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
||||||
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
||||||
// SOFTWARE.
|
|
||||||
//
|
//
|
||||||
|
// Unless required by applicable law or agreed to in writing, software
|
||||||
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
// See the License for the specific language governing permissions and
|
||||||
|
// limitations under the License.
|
||||||
|
|
||||||
#ifndef METROHASH_METROHASH_H
|
#ifndef METROHASH_METROHASH_H
|
||||||
#define METROHASH_METROHASH_H
|
#define METROHASH_METROHASH_H
|
||||||
|
|
||||||
#include <stdint.h>
|
#include "metrohash64.h"
|
||||||
#include <string.h>
|
#include "metrohash128.h"
|
||||||
|
#include "metrohash128crc.h"
|
||||||
// MetroHash 64-bit hash functions
|
|
||||||
void metrohash64_1(const uint8_t * key, uint64_t len, uint32_t seed, uint8_t * out);
|
|
||||||
void metrohash64_2(const uint8_t * key, uint64_t len, uint32_t seed, uint8_t * out);
|
|
||||||
|
|
||||||
// MetroHash 128-bit hash functions
|
|
||||||
void metrohash128_1(const uint8_t * key, uint64_t len, uint32_t seed, uint8_t * out);
|
|
||||||
void metrohash128_2(const uint8_t * key, uint64_t len, uint32_t seed, uint8_t * out);
|
|
||||||
|
|
||||||
// MetroHash 128-bit hash functions using CRC instruction
|
|
||||||
void metrohash128crc_1(const uint8_t * key, uint64_t len, uint32_t seed, uint8_t * out);
|
|
||||||
void metrohash128crc_2(const uint8_t * key, uint64_t len, uint32_t seed, uint8_t * out);
|
|
||||||
|
|
||||||
|
|
||||||
/* rotate right idiom recognized by compiler*/
|
|
||||||
inline static uint64_t rotate_right(uint64_t v, unsigned k)
|
|
||||||
{
|
|
||||||
return (v >> k) | (v << (64 - k));
|
|
||||||
}
|
|
||||||
|
|
||||||
// unaligned reads, fast and safe on Nehalem and later microarchitectures
|
|
||||||
inline static uint64_t read_u64(const void * const ptr)
|
|
||||||
{
|
|
||||||
return static_cast<uint64_t>(*reinterpret_cast<const uint64_t*>(ptr));
|
|
||||||
}
|
|
||||||
|
|
||||||
inline static uint64_t read_u32(const void * const ptr)
|
|
||||||
{
|
|
||||||
return static_cast<uint64_t>(*reinterpret_cast<const uint32_t*>(ptr));
|
|
||||||
}
|
|
||||||
|
|
||||||
inline static uint64_t read_u16(const void * const ptr)
|
|
||||||
{
|
|
||||||
return static_cast<uint64_t>(*reinterpret_cast<const uint16_t*>(ptr));
|
|
||||||
}
|
|
||||||
|
|
||||||
inline static uint64_t read_u8 (const void * const ptr)
|
|
||||||
{
|
|
||||||
return static_cast<uint64_t>(*reinterpret_cast<const uint8_t *>(ptr));
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
#endif // #ifndef METROHASH_METROHASH_H
|
#endif // #ifndef METROHASH_METROHASH_H
|
||||||
|
|||||||
@ -1,29 +1,260 @@
|
|||||||
// metrohash128.cpp
|
// metrohash128.cpp
|
||||||
//
|
//
|
||||||
// The MIT License (MIT)
|
// Copyright 2015-2018 J. Andrew Rogers
|
||||||
//
|
//
|
||||||
// Copyright (c) 2015 J. Andrew Rogers
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
// you may not use this file except in compliance with the License.
|
||||||
|
// You may obtain a copy of the License at
|
||||||
//
|
//
|
||||||
// Permission is hereby granted, free of charge, to any person obtaining a copy
|
// http://www.apache.org/licenses/LICENSE-2.0
|
||||||
// of this software and associated documentation files (the "Software"), to deal
|
|
||||||
// in the Software without restriction, including without limitation the rights
|
|
||||||
// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
||||||
// copies of the Software, and to permit persons to whom the Software is
|
|
||||||
// furnished to do so, subject to the following conditions:
|
|
||||||
//
|
|
||||||
// The above copyright notice and this permission notice shall be included in all
|
|
||||||
// copies or substantial portions of the Software.
|
|
||||||
//
|
|
||||||
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
||||||
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
||||||
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
||||||
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
||||||
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
||||||
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
||||||
// SOFTWARE.
|
|
||||||
//
|
//
|
||||||
|
// Unless required by applicable law or agreed to in writing, software
|
||||||
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
// See the License for the specific language governing permissions and
|
||||||
|
// limitations under the License.
|
||||||
|
|
||||||
|
#include <string.h>
|
||||||
|
#include "platform.h"
|
||||||
|
#include "metrohash128.h"
|
||||||
|
|
||||||
|
const char * MetroHash128::test_string = "012345678901234567890123456789012345678901234567890123456789012";
|
||||||
|
|
||||||
|
const uint8_t MetroHash128::test_seed_0[16] = {
|
||||||
|
0xC7, 0x7C, 0xE2, 0xBF, 0xA4, 0xED, 0x9F, 0x9B,
|
||||||
|
0x05, 0x48, 0xB2, 0xAC, 0x50, 0x74, 0xA2, 0x97
|
||||||
|
};
|
||||||
|
|
||||||
|
const uint8_t MetroHash128::test_seed_1[16] = {
|
||||||
|
0x45, 0xA3, 0xCD, 0xB8, 0x38, 0x19, 0x9D, 0x7F,
|
||||||
|
0xBD, 0xD6, 0x8D, 0x86, 0x7A, 0x14, 0xEC, 0xEF
|
||||||
|
};
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
MetroHash128::MetroHash128(const uint64_t seed)
|
||||||
|
{
|
||||||
|
Initialize(seed);
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
void MetroHash128::Initialize(const uint64_t seed)
|
||||||
|
{
|
||||||
|
// initialize internal hash registers
|
||||||
|
state.v[0] = (static_cast<uint64_t>(seed) - k0) * k3;
|
||||||
|
state.v[1] = (static_cast<uint64_t>(seed) + k1) * k2;
|
||||||
|
state.v[2] = (static_cast<uint64_t>(seed) + k0) * k2;
|
||||||
|
state.v[3] = (static_cast<uint64_t>(seed) - k1) * k3;
|
||||||
|
|
||||||
|
// initialize total length of input
|
||||||
|
bytes = 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
void MetroHash128::Update(const uint8_t * const buffer, const uint64_t length)
|
||||||
|
{
|
||||||
|
const uint8_t * ptr = reinterpret_cast<const uint8_t*>(buffer);
|
||||||
|
const uint8_t * const end = ptr + length;
|
||||||
|
|
||||||
|
// input buffer may be partially filled
|
||||||
|
if (bytes % 32)
|
||||||
|
{
|
||||||
|
uint64_t fill = 32 - (bytes % 32);
|
||||||
|
if (fill > length)
|
||||||
|
fill = length;
|
||||||
|
|
||||||
|
memcpy(input.b + (bytes % 32), ptr, static_cast<size_t>(fill));
|
||||||
|
ptr += fill;
|
||||||
|
bytes += fill;
|
||||||
|
|
||||||
|
// input buffer is still partially filled
|
||||||
|
if ((bytes % 32) != 0) return;
|
||||||
|
|
||||||
|
// process full input buffer
|
||||||
|
state.v[0] += read_u64(&input.b[ 0]) * k0; state.v[0] = rotate_right(state.v[0],29) + state.v[2];
|
||||||
|
state.v[1] += read_u64(&input.b[ 8]) * k1; state.v[1] = rotate_right(state.v[1],29) + state.v[3];
|
||||||
|
state.v[2] += read_u64(&input.b[16]) * k2; state.v[2] = rotate_right(state.v[2],29) + state.v[0];
|
||||||
|
state.v[3] += read_u64(&input.b[24]) * k3; state.v[3] = rotate_right(state.v[3],29) + state.v[1];
|
||||||
|
}
|
||||||
|
|
||||||
|
// bulk update
|
||||||
|
bytes += (end - ptr);
|
||||||
|
while (ptr <= (end - 32))
|
||||||
|
{
|
||||||
|
// process directly from the source, bypassing the input buffer
|
||||||
|
state.v[0] += read_u64(ptr) * k0; ptr += 8; state.v[0] = rotate_right(state.v[0],29) + state.v[2];
|
||||||
|
state.v[1] += read_u64(ptr) * k1; ptr += 8; state.v[1] = rotate_right(state.v[1],29) + state.v[3];
|
||||||
|
state.v[2] += read_u64(ptr) * k2; ptr += 8; state.v[2] = rotate_right(state.v[2],29) + state.v[0];
|
||||||
|
state.v[3] += read_u64(ptr) * k3; ptr += 8; state.v[3] = rotate_right(state.v[3],29) + state.v[1];
|
||||||
|
}
|
||||||
|
|
||||||
|
// store remaining bytes in input buffer
|
||||||
|
if (ptr < end)
|
||||||
|
memcpy(input.b, ptr, end - ptr);
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
void MetroHash128::Finalize(uint8_t * const hash)
|
||||||
|
{
|
||||||
|
// finalize bulk loop, if used
|
||||||
|
if (bytes >= 32)
|
||||||
|
{
|
||||||
|
state.v[2] ^= rotate_right(((state.v[0] + state.v[3]) * k0) + state.v[1], 21) * k1;
|
||||||
|
state.v[3] ^= rotate_right(((state.v[1] + state.v[2]) * k1) + state.v[0], 21) * k0;
|
||||||
|
state.v[0] ^= rotate_right(((state.v[0] + state.v[2]) * k0) + state.v[3], 21) * k1;
|
||||||
|
state.v[1] ^= rotate_right(((state.v[1] + state.v[3]) * k1) + state.v[2], 21) * k0;
|
||||||
|
}
|
||||||
|
|
||||||
|
// process any bytes remaining in the input buffer
|
||||||
|
const uint8_t * ptr = reinterpret_cast<const uint8_t*>(input.b);
|
||||||
|
const uint8_t * const end = ptr + (bytes % 32);
|
||||||
|
|
||||||
|
if ((end - ptr) >= 16)
|
||||||
|
{
|
||||||
|
state.v[0] += read_u64(ptr) * k2; ptr += 8; state.v[0] = rotate_right(state.v[0],33) * k3;
|
||||||
|
state.v[1] += read_u64(ptr) * k2; ptr += 8; state.v[1] = rotate_right(state.v[1],33) * k3;
|
||||||
|
state.v[0] ^= rotate_right((state.v[0] * k2) + state.v[1], 45) * k1;
|
||||||
|
state.v[1] ^= rotate_right((state.v[1] * k3) + state.v[0], 45) * k0;
|
||||||
|
}
|
||||||
|
|
||||||
|
if ((end - ptr) >= 8)
|
||||||
|
{
|
||||||
|
state.v[0] += read_u64(ptr) * k2; ptr += 8; state.v[0] = rotate_right(state.v[0],33) * k3;
|
||||||
|
state.v[0] ^= rotate_right((state.v[0] * k2) + state.v[1], 27) * k1;
|
||||||
|
}
|
||||||
|
|
||||||
|
if ((end - ptr) >= 4)
|
||||||
|
{
|
||||||
|
state.v[1] += read_u32(ptr) * k2; ptr += 4; state.v[1] = rotate_right(state.v[1],33) * k3;
|
||||||
|
state.v[1] ^= rotate_right((state.v[1] * k3) + state.v[0], 46) * k0;
|
||||||
|
}
|
||||||
|
|
||||||
|
if ((end - ptr) >= 2)
|
||||||
|
{
|
||||||
|
state.v[0] += read_u16(ptr) * k2; ptr += 2; state.v[0] = rotate_right(state.v[0],33) * k3;
|
||||||
|
state.v[0] ^= rotate_right((state.v[0] * k2) + state.v[1], 22) * k1;
|
||||||
|
}
|
||||||
|
|
||||||
|
if ((end - ptr) >= 1)
|
||||||
|
{
|
||||||
|
state.v[1] += read_u8 (ptr) * k2; state.v[1] = rotate_right(state.v[1],33) * k3;
|
||||||
|
state.v[1] ^= rotate_right((state.v[1] * k3) + state.v[0], 58) * k0;
|
||||||
|
}
|
||||||
|
|
||||||
|
state.v[0] += rotate_right((state.v[0] * k0) + state.v[1], 13);
|
||||||
|
state.v[1] += rotate_right((state.v[1] * k1) + state.v[0], 37);
|
||||||
|
state.v[0] += rotate_right((state.v[0] * k2) + state.v[1], 13);
|
||||||
|
state.v[1] += rotate_right((state.v[1] * k3) + state.v[0], 37);
|
||||||
|
|
||||||
|
bytes = 0;
|
||||||
|
|
||||||
|
// do any endian conversion here
|
||||||
|
|
||||||
|
memcpy(hash, state.v, 16);
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
void MetroHash128::Hash(const uint8_t * buffer, const uint64_t length, uint8_t * const hash, const uint64_t seed)
|
||||||
|
{
|
||||||
|
const uint8_t * ptr = reinterpret_cast<const uint8_t*>(buffer);
|
||||||
|
const uint8_t * const end = ptr + length;
|
||||||
|
|
||||||
|
uint64_t v[4];
|
||||||
|
|
||||||
|
v[0] = (static_cast<uint64_t>(seed) - k0) * k3;
|
||||||
|
v[1] = (static_cast<uint64_t>(seed) + k1) * k2;
|
||||||
|
|
||||||
|
if (length >= 32)
|
||||||
|
{
|
||||||
|
v[2] = (static_cast<uint64_t>(seed) + k0) * k2;
|
||||||
|
v[3] = (static_cast<uint64_t>(seed) - k1) * k3;
|
||||||
|
|
||||||
|
do
|
||||||
|
{
|
||||||
|
v[0] += read_u64(ptr) * k0; ptr += 8; v[0] = rotate_right(v[0],29) + v[2];
|
||||||
|
v[1] += read_u64(ptr) * k1; ptr += 8; v[1] = rotate_right(v[1],29) + v[3];
|
||||||
|
v[2] += read_u64(ptr) * k2; ptr += 8; v[2] = rotate_right(v[2],29) + v[0];
|
||||||
|
v[3] += read_u64(ptr) * k3; ptr += 8; v[3] = rotate_right(v[3],29) + v[1];
|
||||||
|
}
|
||||||
|
while (ptr <= (end - 32));
|
||||||
|
|
||||||
|
v[2] ^= rotate_right(((v[0] + v[3]) * k0) + v[1], 21) * k1;
|
||||||
|
v[3] ^= rotate_right(((v[1] + v[2]) * k1) + v[0], 21) * k0;
|
||||||
|
v[0] ^= rotate_right(((v[0] + v[2]) * k0) + v[3], 21) * k1;
|
||||||
|
v[1] ^= rotate_right(((v[1] + v[3]) * k1) + v[2], 21) * k0;
|
||||||
|
}
|
||||||
|
|
||||||
|
if ((end - ptr) >= 16)
|
||||||
|
{
|
||||||
|
v[0] += read_u64(ptr) * k2; ptr += 8; v[0] = rotate_right(v[0],33) * k3;
|
||||||
|
v[1] += read_u64(ptr) * k2; ptr += 8; v[1] = rotate_right(v[1],33) * k3;
|
||||||
|
v[0] ^= rotate_right((v[0] * k2) + v[1], 45) * k1;
|
||||||
|
v[1] ^= rotate_right((v[1] * k3) + v[0], 45) * k0;
|
||||||
|
}
|
||||||
|
|
||||||
|
if ((end - ptr) >= 8)
|
||||||
|
{
|
||||||
|
v[0] += read_u64(ptr) * k2; ptr += 8; v[0] = rotate_right(v[0],33) * k3;
|
||||||
|
v[0] ^= rotate_right((v[0] * k2) + v[1], 27) * k1;
|
||||||
|
}
|
||||||
|
|
||||||
|
if ((end - ptr) >= 4)
|
||||||
|
{
|
||||||
|
v[1] += read_u32(ptr) * k2; ptr += 4; v[1] = rotate_right(v[1],33) * k3;
|
||||||
|
v[1] ^= rotate_right((v[1] * k3) + v[0], 46) * k0;
|
||||||
|
}
|
||||||
|
|
||||||
|
if ((end - ptr) >= 2)
|
||||||
|
{
|
||||||
|
v[0] += read_u16(ptr) * k2; ptr += 2; v[0] = rotate_right(v[0],33) * k3;
|
||||||
|
v[0] ^= rotate_right((v[0] * k2) + v[1], 22) * k1;
|
||||||
|
}
|
||||||
|
|
||||||
|
if ((end - ptr) >= 1)
|
||||||
|
{
|
||||||
|
v[1] += read_u8 (ptr) * k2; v[1] = rotate_right(v[1],33) * k3;
|
||||||
|
v[1] ^= rotate_right((v[1] * k3) + v[0], 58) * k0;
|
||||||
|
}
|
||||||
|
|
||||||
|
v[0] += rotate_right((v[0] * k0) + v[1], 13);
|
||||||
|
v[1] += rotate_right((v[1] * k1) + v[0], 37);
|
||||||
|
v[0] += rotate_right((v[0] * k2) + v[1], 13);
|
||||||
|
v[1] += rotate_right((v[1] * k3) + v[0], 37);
|
||||||
|
|
||||||
|
// do any endian conversion here
|
||||||
|
|
||||||
|
memcpy(hash, v, 16);
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
bool MetroHash128::ImplementationVerified()
|
||||||
|
{
|
||||||
|
uint8_t hash[16];
|
||||||
|
const uint8_t * key = reinterpret_cast<const uint8_t *>(MetroHash128::test_string);
|
||||||
|
|
||||||
|
// verify one-shot implementation
|
||||||
|
MetroHash128::Hash(key, strlen(MetroHash128::test_string), hash, 0);
|
||||||
|
if (memcmp(hash, MetroHash128::test_seed_0, 16) != 0) return false;
|
||||||
|
|
||||||
|
MetroHash128::Hash(key, strlen(MetroHash128::test_string), hash, 1);
|
||||||
|
if (memcmp(hash, MetroHash128::test_seed_1, 16) != 0) return false;
|
||||||
|
|
||||||
|
// verify incremental implementation
|
||||||
|
MetroHash128 metro;
|
||||||
|
|
||||||
|
metro.Initialize(0);
|
||||||
|
metro.Update(reinterpret_cast<const uint8_t *>(MetroHash128::test_string), strlen(MetroHash128::test_string));
|
||||||
|
metro.Finalize(hash);
|
||||||
|
if (memcmp(hash, MetroHash128::test_seed_0, 16) != 0) return false;
|
||||||
|
|
||||||
|
metro.Initialize(1);
|
||||||
|
metro.Update(reinterpret_cast<const uint8_t *>(MetroHash128::test_string), strlen(MetroHash128::test_string));
|
||||||
|
metro.Finalize(hash);
|
||||||
|
if (memcmp(hash, MetroHash128::test_seed_1, 16) != 0) return false;
|
||||||
|
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
#include "metrohash.h"
|
|
||||||
|
|
||||||
void metrohash128_1(const uint8_t * key, uint64_t len, uint32_t seed, uint8_t * out)
|
void metrohash128_1(const uint8_t * key, uint64_t len, uint32_t seed, uint8_t * out)
|
||||||
{
|
{
|
||||||
@ -97,6 +328,8 @@ void metrohash128_1(const uint8_t * key, uint64_t len, uint32_t seed, uint8_t *
|
|||||||
v[0] += rotate_right((v[0] * k2) + v[1], 13);
|
v[0] += rotate_right((v[0] * k2) + v[1], 13);
|
||||||
v[1] += rotate_right((v[1] * k3) + v[0], 37);
|
v[1] += rotate_right((v[1] * k3) + v[0], 37);
|
||||||
|
|
||||||
|
// do any endian conversion here
|
||||||
|
|
||||||
memcpy(out, v, 16);
|
memcpy(out, v, 16);
|
||||||
}
|
}
|
||||||
|
|
||||||
@ -173,6 +406,8 @@ void metrohash128_2(const uint8_t * key, uint64_t len, uint32_t seed, uint8_t *
|
|||||||
v[0] += rotate_right((v[0] * k2) + v[1], 33);
|
v[0] += rotate_right((v[0] * k2) + v[1], 33);
|
||||||
v[1] += rotate_right((v[1] * k3) + v[0], 33);
|
v[1] += rotate_right((v[1] * k3) + v[0], 33);
|
||||||
|
|
||||||
|
// do any endian conversion here
|
||||||
|
|
||||||
memcpy(out, v, 16);
|
memcpy(out, v, 16);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
72
contrib/libmetrohash/src/metrohash128.h
Normal file
72
contrib/libmetrohash/src/metrohash128.h
Normal file
@ -0,0 +1,72 @@
|
|||||||
|
// metrohash128.h
|
||||||
|
//
|
||||||
|
// Copyright 2015-2018 J. Andrew Rogers
|
||||||
|
//
|
||||||
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
// you may not use this file except in compliance with the License.
|
||||||
|
// You may obtain a copy of the License at
|
||||||
|
//
|
||||||
|
// http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
//
|
||||||
|
// Unless required by applicable law or agreed to in writing, software
|
||||||
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
// See the License for the specific language governing permissions and
|
||||||
|
// limitations under the License.
|
||||||
|
|
||||||
|
#ifndef METROHASH_METROHASH_128_H
|
||||||
|
#define METROHASH_METROHASH_128_H
|
||||||
|
|
||||||
|
#include <stdint.h>
|
||||||
|
|
||||||
|
class MetroHash128
|
||||||
|
{
|
||||||
|
public:
|
||||||
|
static const uint32_t bits = 128;
|
||||||
|
|
||||||
|
// Constructor initializes the same as Initialize()
|
||||||
|
MetroHash128(const uint64_t seed=0);
|
||||||
|
|
||||||
|
// Initializes internal state for new hash with optional seed
|
||||||
|
void Initialize(const uint64_t seed=0);
|
||||||
|
|
||||||
|
// Update the hash state with a string of bytes. If the length
|
||||||
|
// is sufficiently long, the implementation switches to a bulk
|
||||||
|
// hashing algorithm directly on the argument buffer for speed.
|
||||||
|
void Update(const uint8_t * buffer, const uint64_t length);
|
||||||
|
|
||||||
|
// Constructs the final hash and writes it to the argument buffer.
|
||||||
|
// After a hash is finalized, this instance must be Initialized()-ed
|
||||||
|
// again or the behavior of Update() and Finalize() is undefined.
|
||||||
|
void Finalize(uint8_t * const hash);
|
||||||
|
|
||||||
|
// A non-incremental function implementation. This can be significantly
|
||||||
|
// faster than the incremental implementation for some usage patterns.
|
||||||
|
static void Hash(const uint8_t * buffer, const uint64_t length, uint8_t * const hash, const uint64_t seed=0);
|
||||||
|
|
||||||
|
// Does implementation correctly execute test vectors?
|
||||||
|
static bool ImplementationVerified();
|
||||||
|
|
||||||
|
// test vectors -- Hash(test_string, seed=0) => test_seed_0
|
||||||
|
static const char * test_string;
|
||||||
|
static const uint8_t test_seed_0[16];
|
||||||
|
static const uint8_t test_seed_1[16];
|
||||||
|
|
||||||
|
private:
|
||||||
|
static const uint64_t k0 = 0xC83A91E1;
|
||||||
|
static const uint64_t k1 = 0x8648DBDB;
|
||||||
|
static const uint64_t k2 = 0x7BDEC03B;
|
||||||
|
static const uint64_t k3 = 0x2F5870A5;
|
||||||
|
|
||||||
|
struct { uint64_t v[4]; } state;
|
||||||
|
struct { uint8_t b[32]; } input;
|
||||||
|
uint64_t bytes;
|
||||||
|
};
|
||||||
|
|
||||||
|
|
||||||
|
// Legacy 128-bit hash functions -- do not use
|
||||||
|
void metrohash128_1(const uint8_t * key, uint64_t len, uint32_t seed, uint8_t * out);
|
||||||
|
void metrohash128_2(const uint8_t * key, uint64_t len, uint32_t seed, uint8_t * out);
|
||||||
|
|
||||||
|
|
||||||
|
#endif // #ifndef METROHASH_METROHASH_128_H
|
||||||
@ -1,31 +1,24 @@
|
|||||||
// metrohash128crc.cpp
|
// metrohash128crc.cpp
|
||||||
//
|
//
|
||||||
// The MIT License (MIT)
|
// Copyright 2015-2018 J. Andrew Rogers
|
||||||
//
|
//
|
||||||
// Copyright (c) 2015 J. Andrew Rogers
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
// you may not use this file except in compliance with the License.
|
||||||
|
// You may obtain a copy of the License at
|
||||||
//
|
//
|
||||||
// Permission is hereby granted, free of charge, to any person obtaining a copy
|
// http://www.apache.org/licenses/LICENSE-2.0
|
||||||
// of this software and associated documentation files (the "Software"), to deal
|
|
||||||
// in the Software without restriction, including without limitation the rights
|
|
||||||
// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
||||||
// copies of the Software, and to permit persons to whom the Software is
|
|
||||||
// furnished to do so, subject to the following conditions:
|
|
||||||
//
|
|
||||||
// The above copyright notice and this permission notice shall be included in all
|
|
||||||
// copies or substantial portions of the Software.
|
|
||||||
//
|
|
||||||
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
||||||
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
||||||
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
||||||
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
||||||
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
||||||
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
||||||
// SOFTWARE.
|
|
||||||
//
|
//
|
||||||
|
// Unless required by applicable law or agreed to in writing, software
|
||||||
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
// See the License for the specific language governing permissions and
|
||||||
|
// limitations under the License.
|
||||||
|
|
||||||
|
|
||||||
#include "metrohash.h"
|
|
||||||
#include <nmmintrin.h>
|
#include <nmmintrin.h>
|
||||||
|
#include <string.h>
|
||||||
|
#include "metrohash.h"
|
||||||
|
#include "platform.h"
|
||||||
|
|
||||||
|
|
||||||
void metrohash128crc_1(const uint8_t * key, uint64_t len, uint32_t seed, uint8_t * out)
|
void metrohash128crc_1(const uint8_t * key, uint64_t len, uint32_t seed, uint8_t * out)
|
||||||
|
|||||||
27
contrib/libmetrohash/src/metrohash128crc.h
Normal file
27
contrib/libmetrohash/src/metrohash128crc.h
Normal file
@ -0,0 +1,27 @@
|
|||||||
|
// metrohash128crc.h
|
||||||
|
//
|
||||||
|
// Copyright 2015-2018 J. Andrew Rogers
|
||||||
|
//
|
||||||
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
// you may not use this file except in compliance with the License.
|
||||||
|
// You may obtain a copy of the License at
|
||||||
|
//
|
||||||
|
// http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
//
|
||||||
|
// Unless required by applicable law or agreed to in writing, software
|
||||||
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
// See the License for the specific language governing permissions and
|
||||||
|
// limitations under the License.
|
||||||
|
|
||||||
|
#ifndef METROHASH_METROHASH_128_CRC_H
|
||||||
|
#define METROHASH_METROHASH_128_CRC_H
|
||||||
|
|
||||||
|
#include <stdint.h>
|
||||||
|
|
||||||
|
// Legacy 128-bit hash functions
|
||||||
|
void metrohash128crc_1(const uint8_t * key, uint64_t len, uint32_t seed, uint8_t * out);
|
||||||
|
void metrohash128crc_2(const uint8_t * key, uint64_t len, uint32_t seed, uint8_t * out);
|
||||||
|
|
||||||
|
|
||||||
|
#endif // #ifndef METROHASH_METROHASH_128_CRC_H
|
||||||
@ -1,29 +1,257 @@
|
|||||||
// metrohash64.cpp
|
// metrohash64.cpp
|
||||||
//
|
//
|
||||||
// The MIT License (MIT)
|
// Copyright 2015-2018 J. Andrew Rogers
|
||||||
//
|
//
|
||||||
// Copyright (c) 2015 J. Andrew Rogers
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
// you may not use this file except in compliance with the License.
|
||||||
|
// You may obtain a copy of the License at
|
||||||
//
|
//
|
||||||
// Permission is hereby granted, free of charge, to any person obtaining a copy
|
// http://www.apache.org/licenses/LICENSE-2.0
|
||||||
// of this software and associated documentation files (the "Software"), to deal
|
|
||||||
// in the Software without restriction, including without limitation the rights
|
|
||||||
// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
||||||
// copies of the Software, and to permit persons to whom the Software is
|
|
||||||
// furnished to do so, subject to the following conditions:
|
|
||||||
//
|
|
||||||
// The above copyright notice and this permission notice shall be included in all
|
|
||||||
// copies or substantial portions of the Software.
|
|
||||||
//
|
|
||||||
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
||||||
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
||||||
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
||||||
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
||||||
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
||||||
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
||||||
// SOFTWARE.
|
|
||||||
//
|
//
|
||||||
|
// Unless required by applicable law or agreed to in writing, software
|
||||||
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
// See the License for the specific language governing permissions and
|
||||||
|
// limitations under the License.
|
||||||
|
|
||||||
|
#include "platform.h"
|
||||||
|
#include "metrohash64.h"
|
||||||
|
|
||||||
|
#include <cstring>
|
||||||
|
|
||||||
|
const char * MetroHash64::test_string = "012345678901234567890123456789012345678901234567890123456789012";
|
||||||
|
|
||||||
|
const uint8_t MetroHash64::test_seed_0[8] = { 0x6B, 0x75, 0x3D, 0xAE, 0x06, 0x70, 0x4B, 0xAD };
|
||||||
|
const uint8_t MetroHash64::test_seed_1[8] = { 0x3B, 0x0D, 0x48, 0x1C, 0xF4, 0xB9, 0xB8, 0xDF };
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
MetroHash64::MetroHash64(const uint64_t seed)
|
||||||
|
{
|
||||||
|
Initialize(seed);
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
void MetroHash64::Initialize(const uint64_t seed)
|
||||||
|
{
|
||||||
|
vseed = (static_cast<uint64_t>(seed) + k2) * k0;
|
||||||
|
|
||||||
|
// initialize internal hash registers
|
||||||
|
state.v[0] = vseed;
|
||||||
|
state.v[1] = vseed;
|
||||||
|
state.v[2] = vseed;
|
||||||
|
state.v[3] = vseed;
|
||||||
|
|
||||||
|
// initialize total length of input
|
||||||
|
bytes = 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
void MetroHash64::Update(const uint8_t * const buffer, const uint64_t length)
|
||||||
|
{
|
||||||
|
const uint8_t * ptr = reinterpret_cast<const uint8_t*>(buffer);
|
||||||
|
const uint8_t * const end = ptr + length;
|
||||||
|
|
||||||
|
// input buffer may be partially filled
|
||||||
|
if (bytes % 32)
|
||||||
|
{
|
||||||
|
uint64_t fill = 32 - (bytes % 32);
|
||||||
|
if (fill > length)
|
||||||
|
fill = length;
|
||||||
|
|
||||||
|
memcpy(input.b + (bytes % 32), ptr, static_cast<size_t>(fill));
|
||||||
|
ptr += fill;
|
||||||
|
bytes += fill;
|
||||||
|
|
||||||
|
// input buffer is still partially filled
|
||||||
|
if ((bytes % 32) != 0) return;
|
||||||
|
|
||||||
|
// process full input buffer
|
||||||
|
state.v[0] += read_u64(&input.b[ 0]) * k0; state.v[0] = rotate_right(state.v[0],29) + state.v[2];
|
||||||
|
state.v[1] += read_u64(&input.b[ 8]) * k1; state.v[1] = rotate_right(state.v[1],29) + state.v[3];
|
||||||
|
state.v[2] += read_u64(&input.b[16]) * k2; state.v[2] = rotate_right(state.v[2],29) + state.v[0];
|
||||||
|
state.v[3] += read_u64(&input.b[24]) * k3; state.v[3] = rotate_right(state.v[3],29) + state.v[1];
|
||||||
|
}
|
||||||
|
|
||||||
|
// bulk update
|
||||||
|
bytes += static_cast<uint64_t>(end - ptr);
|
||||||
|
while (ptr <= (end - 32))
|
||||||
|
{
|
||||||
|
// process directly from the source, bypassing the input buffer
|
||||||
|
state.v[0] += read_u64(ptr) * k0; ptr += 8; state.v[0] = rotate_right(state.v[0],29) + state.v[2];
|
||||||
|
state.v[1] += read_u64(ptr) * k1; ptr += 8; state.v[1] = rotate_right(state.v[1],29) + state.v[3];
|
||||||
|
state.v[2] += read_u64(ptr) * k2; ptr += 8; state.v[2] = rotate_right(state.v[2],29) + state.v[0];
|
||||||
|
state.v[3] += read_u64(ptr) * k3; ptr += 8; state.v[3] = rotate_right(state.v[3],29) + state.v[1];
|
||||||
|
}
|
||||||
|
|
||||||
|
// store remaining bytes in input buffer
|
||||||
|
if (ptr < end)
|
||||||
|
memcpy(input.b, ptr, static_cast<size_t>(end - ptr));
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
void MetroHash64::Finalize(uint8_t * const hash)
|
||||||
|
{
|
||||||
|
// finalize bulk loop, if used
|
||||||
|
if (bytes >= 32)
|
||||||
|
{
|
||||||
|
state.v[2] ^= rotate_right(((state.v[0] + state.v[3]) * k0) + state.v[1], 37) * k1;
|
||||||
|
state.v[3] ^= rotate_right(((state.v[1] + state.v[2]) * k1) + state.v[0], 37) * k0;
|
||||||
|
state.v[0] ^= rotate_right(((state.v[0] + state.v[2]) * k0) + state.v[3], 37) * k1;
|
||||||
|
state.v[1] ^= rotate_right(((state.v[1] + state.v[3]) * k1) + state.v[2], 37) * k0;
|
||||||
|
|
||||||
|
state.v[0] = vseed + (state.v[0] ^ state.v[1]);
|
||||||
|
}
|
||||||
|
|
||||||
|
// process any bytes remaining in the input buffer
|
||||||
|
const uint8_t * ptr = reinterpret_cast<const uint8_t*>(input.b);
|
||||||
|
const uint8_t * const end = ptr + (bytes % 32);
|
||||||
|
|
||||||
|
if ((end - ptr) >= 16)
|
||||||
|
{
|
||||||
|
state.v[1] = state.v[0] + (read_u64(ptr) * k2); ptr += 8; state.v[1] = rotate_right(state.v[1],29) * k3;
|
||||||
|
state.v[2] = state.v[0] + (read_u64(ptr) * k2); ptr += 8; state.v[2] = rotate_right(state.v[2],29) * k3;
|
||||||
|
state.v[1] ^= rotate_right(state.v[1] * k0, 21) + state.v[2];
|
||||||
|
state.v[2] ^= rotate_right(state.v[2] * k3, 21) + state.v[1];
|
||||||
|
state.v[0] += state.v[2];
|
||||||
|
}
|
||||||
|
|
||||||
|
if ((end - ptr) >= 8)
|
||||||
|
{
|
||||||
|
state.v[0] += read_u64(ptr) * k3; ptr += 8;
|
||||||
|
state.v[0] ^= rotate_right(state.v[0], 55) * k1;
|
||||||
|
}
|
||||||
|
|
||||||
|
if ((end - ptr) >= 4)
|
||||||
|
{
|
||||||
|
state.v[0] += read_u32(ptr) * k3; ptr += 4;
|
||||||
|
state.v[0] ^= rotate_right(state.v[0], 26) * k1;
|
||||||
|
}
|
||||||
|
|
||||||
|
if ((end - ptr) >= 2)
|
||||||
|
{
|
||||||
|
state.v[0] += read_u16(ptr) * k3; ptr += 2;
|
||||||
|
state.v[0] ^= rotate_right(state.v[0], 48) * k1;
|
||||||
|
}
|
||||||
|
|
||||||
|
if ((end - ptr) >= 1)
|
||||||
|
{
|
||||||
|
state.v[0] += read_u8 (ptr) * k3;
|
||||||
|
state.v[0] ^= rotate_right(state.v[0], 37) * k1;
|
||||||
|
}
|
||||||
|
|
||||||
|
state.v[0] ^= rotate_right(state.v[0], 28);
|
||||||
|
state.v[0] *= k0;
|
||||||
|
state.v[0] ^= rotate_right(state.v[0], 29);
|
||||||
|
|
||||||
|
bytes = 0;
|
||||||
|
|
||||||
|
// do any endian conversion here
|
||||||
|
|
||||||
|
memcpy(hash, state.v, 8);
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
void MetroHash64::Hash(const uint8_t * buffer, const uint64_t length, uint8_t * const hash, const uint64_t seed)
|
||||||
|
{
|
||||||
|
const uint8_t * ptr = reinterpret_cast<const uint8_t*>(buffer);
|
||||||
|
const uint8_t * const end = ptr + length;
|
||||||
|
|
||||||
|
uint64_t h = (static_cast<uint64_t>(seed) + k2) * k0;
|
||||||
|
|
||||||
|
if (length >= 32)
|
||||||
|
{
|
||||||
|
uint64_t v[4];
|
||||||
|
v[0] = h;
|
||||||
|
v[1] = h;
|
||||||
|
v[2] = h;
|
||||||
|
v[3] = h;
|
||||||
|
|
||||||
|
do
|
||||||
|
{
|
||||||
|
v[0] += read_u64(ptr) * k0; ptr += 8; v[0] = rotate_right(v[0],29) + v[2];
|
||||||
|
v[1] += read_u64(ptr) * k1; ptr += 8; v[1] = rotate_right(v[1],29) + v[3];
|
||||||
|
v[2] += read_u64(ptr) * k2; ptr += 8; v[2] = rotate_right(v[2],29) + v[0];
|
||||||
|
v[3] += read_u64(ptr) * k3; ptr += 8; v[3] = rotate_right(v[3],29) + v[1];
|
||||||
|
}
|
||||||
|
while (ptr <= (end - 32));
|
||||||
|
|
||||||
|
v[2] ^= rotate_right(((v[0] + v[3]) * k0) + v[1], 37) * k1;
|
||||||
|
v[3] ^= rotate_right(((v[1] + v[2]) * k1) + v[0], 37) * k0;
|
||||||
|
v[0] ^= rotate_right(((v[0] + v[2]) * k0) + v[3], 37) * k1;
|
||||||
|
v[1] ^= rotate_right(((v[1] + v[3]) * k1) + v[2], 37) * k0;
|
||||||
|
h += v[0] ^ v[1];
|
||||||
|
}
|
||||||
|
|
||||||
|
if ((end - ptr) >= 16)
|
||||||
|
{
|
||||||
|
uint64_t v0 = h + (read_u64(ptr) * k2); ptr += 8; v0 = rotate_right(v0,29) * k3;
|
||||||
|
uint64_t v1 = h + (read_u64(ptr) * k2); ptr += 8; v1 = rotate_right(v1,29) * k3;
|
||||||
|
v0 ^= rotate_right(v0 * k0, 21) + v1;
|
||||||
|
v1 ^= rotate_right(v1 * k3, 21) + v0;
|
||||||
|
h += v1;
|
||||||
|
}
|
||||||
|
|
||||||
|
if ((end - ptr) >= 8)
|
||||||
|
{
|
||||||
|
h += read_u64(ptr) * k3; ptr += 8;
|
||||||
|
h ^= rotate_right(h, 55) * k1;
|
||||||
|
}
|
||||||
|
|
||||||
|
if ((end - ptr) >= 4)
|
||||||
|
{
|
||||||
|
h += read_u32(ptr) * k3; ptr += 4;
|
||||||
|
h ^= rotate_right(h, 26) * k1;
|
||||||
|
}
|
||||||
|
|
||||||
|
if ((end - ptr) >= 2)
|
||||||
|
{
|
||||||
|
h += read_u16(ptr) * k3; ptr += 2;
|
||||||
|
h ^= rotate_right(h, 48) * k1;
|
||||||
|
}
|
||||||
|
|
||||||
|
if ((end - ptr) >= 1)
|
||||||
|
{
|
||||||
|
h += read_u8 (ptr) * k3;
|
||||||
|
h ^= rotate_right(h, 37) * k1;
|
||||||
|
}
|
||||||
|
|
||||||
|
h ^= rotate_right(h, 28);
|
||||||
|
h *= k0;
|
||||||
|
h ^= rotate_right(h, 29);
|
||||||
|
|
||||||
|
memcpy(hash, &h, 8);
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
bool MetroHash64::ImplementationVerified()
|
||||||
|
{
|
||||||
|
uint8_t hash[8];
|
||||||
|
const uint8_t * key = reinterpret_cast<const uint8_t *>(MetroHash64::test_string);
|
||||||
|
|
||||||
|
// verify one-shot implementation
|
||||||
|
MetroHash64::Hash(key, strlen(MetroHash64::test_string), hash, 0);
|
||||||
|
if (memcmp(hash, MetroHash64::test_seed_0, 8) != 0) return false;
|
||||||
|
|
||||||
|
MetroHash64::Hash(key, strlen(MetroHash64::test_string), hash, 1);
|
||||||
|
if (memcmp(hash, MetroHash64::test_seed_1, 8) != 0) return false;
|
||||||
|
|
||||||
|
// verify incremental implementation
|
||||||
|
MetroHash64 metro;
|
||||||
|
|
||||||
|
metro.Initialize(0);
|
||||||
|
metro.Update(reinterpret_cast<const uint8_t *>(MetroHash64::test_string), strlen(MetroHash64::test_string));
|
||||||
|
metro.Finalize(hash);
|
||||||
|
if (memcmp(hash, MetroHash64::test_seed_0, 8) != 0) return false;
|
||||||
|
|
||||||
|
metro.Initialize(1);
|
||||||
|
metro.Update(reinterpret_cast<const uint8_t *>(MetroHash64::test_string), strlen(MetroHash64::test_string));
|
||||||
|
metro.Finalize(hash);
|
||||||
|
if (memcmp(hash, MetroHash64::test_seed_1, 8) != 0) return false;
|
||||||
|
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
#include "metrohash.h"
|
|
||||||
|
|
||||||
void metrohash64_1(const uint8_t * key, uint64_t len, uint32_t seed, uint8_t * out)
|
void metrohash64_1(const uint8_t * key, uint64_t len, uint32_t seed, uint8_t * out)
|
||||||
{
|
{
|
||||||
|
|||||||
73
contrib/libmetrohash/src/metrohash64.h
Normal file
73
contrib/libmetrohash/src/metrohash64.h
Normal file
@ -0,0 +1,73 @@
|
|||||||
|
// metrohash64.h
|
||||||
|
//
|
||||||
|
// Copyright 2015-2018 J. Andrew Rogers
|
||||||
|
//
|
||||||
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
// you may not use this file except in compliance with the License.
|
||||||
|
// You may obtain a copy of the License at
|
||||||
|
//
|
||||||
|
// http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
//
|
||||||
|
// Unless required by applicable law or agreed to in writing, software
|
||||||
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
// See the License for the specific language governing permissions and
|
||||||
|
// limitations under the License.
|
||||||
|
|
||||||
|
#ifndef METROHASH_METROHASH_64_H
|
||||||
|
#define METROHASH_METROHASH_64_H
|
||||||
|
|
||||||
|
#include <stdint.h>
|
||||||
|
|
||||||
|
class MetroHash64
|
||||||
|
{
|
||||||
|
public:
|
||||||
|
static const uint32_t bits = 64;
|
||||||
|
|
||||||
|
// Constructor initializes the same as Initialize()
|
||||||
|
MetroHash64(const uint64_t seed=0);
|
||||||
|
|
||||||
|
// Initializes internal state for new hash with optional seed
|
||||||
|
void Initialize(const uint64_t seed=0);
|
||||||
|
|
||||||
|
// Update the hash state with a string of bytes. If the length
|
||||||
|
// is sufficiently long, the implementation switches to a bulk
|
||||||
|
// hashing algorithm directly on the argument buffer for speed.
|
||||||
|
void Update(const uint8_t * buffer, const uint64_t length);
|
||||||
|
|
||||||
|
// Constructs the final hash and writes it to the argument buffer.
|
||||||
|
// After a hash is finalized, this instance must be Initialized()-ed
|
||||||
|
// again or the behavior of Update() and Finalize() is undefined.
|
||||||
|
void Finalize(uint8_t * const hash);
|
||||||
|
|
||||||
|
// A non-incremental function implementation. This can be significantly
|
||||||
|
// faster than the incremental implementation for some usage patterns.
|
||||||
|
static void Hash(const uint8_t * buffer, const uint64_t length, uint8_t * const hash, const uint64_t seed=0);
|
||||||
|
|
||||||
|
// Does implementation correctly execute test vectors?
|
||||||
|
static bool ImplementationVerified();
|
||||||
|
|
||||||
|
// test vectors -- Hash(test_string, seed=0) => test_seed_0
|
||||||
|
static const char * test_string;
|
||||||
|
static const uint8_t test_seed_0[8];
|
||||||
|
static const uint8_t test_seed_1[8];
|
||||||
|
|
||||||
|
private:
|
||||||
|
static const uint64_t k0 = 0xD6D018F5;
|
||||||
|
static const uint64_t k1 = 0xA2AA033B;
|
||||||
|
static const uint64_t k2 = 0x62992FC1;
|
||||||
|
static const uint64_t k3 = 0x30BC5B29;
|
||||||
|
|
||||||
|
struct { uint64_t v[4]; } state;
|
||||||
|
struct { uint8_t b[32]; } input;
|
||||||
|
uint64_t bytes;
|
||||||
|
uint64_t vseed;
|
||||||
|
};
|
||||||
|
|
||||||
|
|
||||||
|
// Legacy 64-bit hash functions -- do not use
|
||||||
|
void metrohash64_1(const uint8_t * key, uint64_t len, uint32_t seed, uint8_t * out);
|
||||||
|
void metrohash64_2(const uint8_t * key, uint64_t len, uint32_t seed, uint8_t * out);
|
||||||
|
|
||||||
|
|
||||||
|
#endif // #ifndef METROHASH_METROHASH_64_H
|
||||||
50
contrib/libmetrohash/src/platform.h
Normal file
50
contrib/libmetrohash/src/platform.h
Normal file
@ -0,0 +1,50 @@
|
|||||||
|
// platform.h
|
||||||
|
//
|
||||||
|
// Copyright 2015-2018 J. Andrew Rogers
|
||||||
|
//
|
||||||
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
// you may not use this file except in compliance with the License.
|
||||||
|
// You may obtain a copy of the License at
|
||||||
|
//
|
||||||
|
// http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
//
|
||||||
|
// Unless required by applicable law or agreed to in writing, software
|
||||||
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
// See the License for the specific language governing permissions and
|
||||||
|
// limitations under the License.
|
||||||
|
|
||||||
|
#ifndef METROHASH_PLATFORM_H
|
||||||
|
#define METROHASH_PLATFORM_H
|
||||||
|
|
||||||
|
#include <stdint.h>
|
||||||
|
|
||||||
|
// rotate right idiom recognized by most compilers
|
||||||
|
inline static uint64_t rotate_right(uint64_t v, unsigned k)
|
||||||
|
{
|
||||||
|
return (v >> k) | (v << (64 - k));
|
||||||
|
}
|
||||||
|
|
||||||
|
// unaligned reads, fast and safe on Nehalem and later microarchitectures
|
||||||
|
inline static uint64_t read_u64(const void * const ptr)
|
||||||
|
{
|
||||||
|
return static_cast<uint64_t>(*reinterpret_cast<const uint64_t*>(ptr));
|
||||||
|
}
|
||||||
|
|
||||||
|
inline static uint64_t read_u32(const void * const ptr)
|
||||||
|
{
|
||||||
|
return static_cast<uint64_t>(*reinterpret_cast<const uint32_t*>(ptr));
|
||||||
|
}
|
||||||
|
|
||||||
|
inline static uint64_t read_u16(const void * const ptr)
|
||||||
|
{
|
||||||
|
return static_cast<uint64_t>(*reinterpret_cast<const uint16_t*>(ptr));
|
||||||
|
}
|
||||||
|
|
||||||
|
inline static uint64_t read_u8 (const void * const ptr)
|
||||||
|
{
|
||||||
|
return static_cast<uint64_t>(*reinterpret_cast<const uint8_t *>(ptr));
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
#endif // #ifndef METROHASH_PLATFORM_H
|
||||||
@ -1,27 +1,18 @@
|
|||||||
// testvector.h
|
// testvector.h
|
||||||
//
|
//
|
||||||
// The MIT License (MIT)
|
// Copyright 2015-2018 J. Andrew Rogers
|
||||||
//
|
//
|
||||||
// Copyright (c) 2015 J. Andrew Rogers
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
// you may not use this file except in compliance with the License.
|
||||||
|
// You may obtain a copy of the License at
|
||||||
//
|
//
|
||||||
// Permission is hereby granted, free of charge, to any person obtaining a copy
|
// http://www.apache.org/licenses/LICENSE-2.0
|
||||||
// of this software and associated documentation files (the "Software"), to deal
|
|
||||||
// in the Software without restriction, including without limitation the rights
|
|
||||||
// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
||||||
// copies of the Software, and to permit persons to whom the Software is
|
|
||||||
// furnished to do so, subject to the following conditions:
|
|
||||||
//
|
|
||||||
// The above copyright notice and this permission notice shall be included in all
|
|
||||||
// copies or substantial portions of the Software.
|
|
||||||
//
|
|
||||||
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
||||||
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
||||||
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
||||||
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
||||||
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
||||||
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
||||||
// SOFTWARE.
|
|
||||||
//
|
//
|
||||||
|
// Unless required by applicable law or agreed to in writing, software
|
||||||
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
// See the License for the specific language governing permissions and
|
||||||
|
// limitations under the License.
|
||||||
|
|
||||||
#ifndef METROHASH_TESTVECTOR_H
|
#ifndef METROHASH_TESTVECTOR_H
|
||||||
#define METROHASH_TESTVECTOR_H
|
#define METROHASH_TESTVECTOR_H
|
||||||
@ -46,6 +37,8 @@ struct TestVectorData
|
|||||||
|
|
||||||
static const char * test_key_63 = "012345678901234567890123456789012345678901234567890123456789012";
|
static const char * test_key_63 = "012345678901234567890123456789012345678901234567890123456789012";
|
||||||
|
|
||||||
|
// The hash assumes a little-endian architecture. Treating the hash results
|
||||||
|
// as an array of uint64_t should enable conversion for big-endian implementations.
|
||||||
const TestVectorData TestVector [] =
|
const TestVectorData TestVector [] =
|
||||||
{
|
{
|
||||||
// seed = 0
|
// seed = 0
|
||||||
|
|||||||
2
contrib/librdkafka
vendored
2
contrib/librdkafka
vendored
@ -1 +1 @@
|
|||||||
Subproject commit 7478b5ef16aadd6543fe38bc6a2deb895c70da98
|
Subproject commit 51ae5f5fd8b742e56f47a8bb0136344868818285
|
||||||
@ -2,18 +2,21 @@ set(RDKAFKA_SOURCE_DIR ${CMAKE_SOURCE_DIR}/contrib/librdkafka/src)
|
|||||||
|
|
||||||
set(SRCS
|
set(SRCS
|
||||||
${RDKAFKA_SOURCE_DIR}/crc32c.c
|
${RDKAFKA_SOURCE_DIR}/crc32c.c
|
||||||
|
${RDKAFKA_SOURCE_DIR}/rdkafka_zstd.c
|
||||||
${RDKAFKA_SOURCE_DIR}/rdaddr.c
|
${RDKAFKA_SOURCE_DIR}/rdaddr.c
|
||||||
${RDKAFKA_SOURCE_DIR}/rdavl.c
|
${RDKAFKA_SOURCE_DIR}/rdavl.c
|
||||||
${RDKAFKA_SOURCE_DIR}/rdbuf.c
|
${RDKAFKA_SOURCE_DIR}/rdbuf.c
|
||||||
${RDKAFKA_SOURCE_DIR}/rdcrc32.c
|
${RDKAFKA_SOURCE_DIR}/rdcrc32.c
|
||||||
${RDKAFKA_SOURCE_DIR}/rdkafka.c
|
${RDKAFKA_SOURCE_DIR}/rdkafka.c
|
||||||
${RDKAFKA_SOURCE_DIR}/rdkafka_assignor.c
|
${RDKAFKA_SOURCE_DIR}/rdkafka_assignor.c
|
||||||
|
${RDKAFKA_SOURCE_DIR}/rdkafka_background.c
|
||||||
${RDKAFKA_SOURCE_DIR}/rdkafka_broker.c
|
${RDKAFKA_SOURCE_DIR}/rdkafka_broker.c
|
||||||
${RDKAFKA_SOURCE_DIR}/rdkafka_buf.c
|
${RDKAFKA_SOURCE_DIR}/rdkafka_buf.c
|
||||||
${RDKAFKA_SOURCE_DIR}/rdkafka_cgrp.c
|
${RDKAFKA_SOURCE_DIR}/rdkafka_cgrp.c
|
||||||
${RDKAFKA_SOURCE_DIR}/rdkafka_conf.c
|
${RDKAFKA_SOURCE_DIR}/rdkafka_conf.c
|
||||||
${RDKAFKA_SOURCE_DIR}/rdkafka_event.c
|
${RDKAFKA_SOURCE_DIR}/rdkafka_event.c
|
||||||
${RDKAFKA_SOURCE_DIR}/rdkafka_feature.c
|
${RDKAFKA_SOURCE_DIR}/rdkafka_feature.c
|
||||||
|
${RDKAFKA_SOURCE_DIR}/rdkafka_idempotence.c
|
||||||
${RDKAFKA_SOURCE_DIR}/rdkafka_lz4.c
|
${RDKAFKA_SOURCE_DIR}/rdkafka_lz4.c
|
||||||
${RDKAFKA_SOURCE_DIR}/rdkafka_metadata.c
|
${RDKAFKA_SOURCE_DIR}/rdkafka_metadata.c
|
||||||
${RDKAFKA_SOURCE_DIR}/rdkafka_metadata_cache.c
|
${RDKAFKA_SOURCE_DIR}/rdkafka_metadata_cache.c
|
||||||
@ -47,14 +50,12 @@ ${RDKAFKA_SOURCE_DIR}/rdunittest.c
|
|||||||
${RDKAFKA_SOURCE_DIR}/rdvarint.c
|
${RDKAFKA_SOURCE_DIR}/rdvarint.c
|
||||||
${RDKAFKA_SOURCE_DIR}/snappy.c
|
${RDKAFKA_SOURCE_DIR}/snappy.c
|
||||||
${RDKAFKA_SOURCE_DIR}/tinycthread.c
|
${RDKAFKA_SOURCE_DIR}/tinycthread.c
|
||||||
${RDKAFKA_SOURCE_DIR}/xxhash.c
|
${RDKAFKA_SOURCE_DIR}/tinycthread_extra.c
|
||||||
${RDKAFKA_SOURCE_DIR}/lz4.c
|
|
||||||
${RDKAFKA_SOURCE_DIR}/lz4frame.c
|
|
||||||
${RDKAFKA_SOURCE_DIR}/lz4hc.c
|
|
||||||
${RDKAFKA_SOURCE_DIR}/rdgz.c
|
${RDKAFKA_SOURCE_DIR}/rdgz.c
|
||||||
)
|
)
|
||||||
|
|
||||||
add_library(rdkafka ${LINK_MODE} ${SRCS})
|
add_library(rdkafka ${LINK_MODE} ${SRCS})
|
||||||
target_include_directories(rdkafka PRIVATE include)
|
target_include_directories(rdkafka SYSTEM PUBLIC include)
|
||||||
target_include_directories(rdkafka SYSTEM PUBLIC ${RDKAFKA_SOURCE_DIR})
|
target_include_directories(rdkafka SYSTEM PUBLIC ${RDKAFKA_SOURCE_DIR}) # Because weird logic with "include_next" is used.
|
||||||
target_link_libraries(rdkafka PUBLIC ${ZLIB_LIBRARIES} ${OPENSSL_SSL_LIBRARY} ${OPENSSL_CRYPTO_LIBRARY})
|
target_include_directories(rdkafka SYSTEM PRIVATE ${ZSTD_INCLUDE_DIR}/common) # Because wrong path to "zstd_errors.h" is used.
|
||||||
|
target_link_libraries(rdkafka PUBLIC ${ZLIB_LIBRARIES} ${ZSTD_LIBRARY} ${LZ4_LIBRARY} ${OPENSSL_SSL_LIBRARY} ${OPENSSL_CRYPTO_LIBRARY})
|
||||||
|
|||||||
@ -51,6 +51,8 @@
|
|||||||
//#define WITH_PLUGINS 1
|
//#define WITH_PLUGINS 1
|
||||||
// zlib
|
// zlib
|
||||||
#define WITH_ZLIB 1
|
#define WITH_ZLIB 1
|
||||||
|
// zstd
|
||||||
|
#define WITH_ZSTD 1
|
||||||
// WITH_SNAPPY
|
// WITH_SNAPPY
|
||||||
#define WITH_SNAPPY 1
|
#define WITH_SNAPPY 1
|
||||||
// WITH_SOCKEM
|
// WITH_SOCKEM
|
||||||
@ -60,7 +62,9 @@
|
|||||||
// WITH_SASL_SCRAM
|
// WITH_SASL_SCRAM
|
||||||
//#define WITH_SASL_SCRAM 1
|
//#define WITH_SASL_SCRAM 1
|
||||||
// crc32chw
|
// crc32chw
|
||||||
|
#if !defined(__PPC__)
|
||||||
#define WITH_CRC32C_HW 1
|
#define WITH_CRC32C_HW 1
|
||||||
|
#endif
|
||||||
// regex
|
// regex
|
||||||
#define HAVE_REGEX 1
|
#define HAVE_REGEX 1
|
||||||
// strndup
|
// strndup
|
||||||
@ -71,4 +75,8 @@
|
|||||||
#define HAVE_PTHREAD_SETNAME_GNU 1
|
#define HAVE_PTHREAD_SETNAME_GNU 1
|
||||||
// python
|
// python
|
||||||
//#define HAVE_PYTHON 1
|
//#define HAVE_PYTHON 1
|
||||||
|
// C11 threads
|
||||||
|
#if (__STDC_VERSION__ >= 201112L) && !defined(__STDC_NO_THREADS__)
|
||||||
|
# define WITH_C11THREADS 1
|
||||||
|
#endif
|
||||||
#endif /* _CONFIG_H_ */
|
#endif /* _CONFIG_H_ */
|
||||||
|
|||||||
5
contrib/librdkafka-cmake/include/librdkafka/rdkafka.h
Normal file
5
contrib/librdkafka-cmake/include/librdkafka/rdkafka.h
Normal file
@ -0,0 +1,5 @@
|
|||||||
|
#if __has_include(<rdkafka.h>) // maybe bundled
|
||||||
|
# include_next <rdkafka.h> // Y_IGNORE
|
||||||
|
#else // system
|
||||||
|
# include_next <librdkafka/rdkafka.h>
|
||||||
|
#endif
|
||||||
@ -1,13 +1,11 @@
|
|||||||
|
|
||||||
enable_language(ASM)
|
enable_language(ASM)
|
||||||
|
|
||||||
|
if (ARCH_PPC64LE)
|
||||||
add_library(unwind
|
add_library(unwind
|
||||||
src/mi/init.c
|
src/mi/init.c
|
||||||
src/mi/flush_cache.c
|
src/mi/flush_cache.c
|
||||||
src/mi/mempool.c
|
src/mi/mempool.c
|
||||||
src/mi/strerror.c
|
src/mi/strerror.c
|
||||||
src/x86_64/is_fpreg.c
|
|
||||||
src/x86_64/regname.c
|
|
||||||
src/mi/_ReadULEB.c
|
src/mi/_ReadULEB.c
|
||||||
src/mi/_ReadSLEB.c
|
src/mi/_ReadSLEB.c
|
||||||
src/mi/backtrace.c
|
src/mi/backtrace.c
|
||||||
@ -26,6 +24,70 @@ src/mi/Lset_reg.c
|
|||||||
src/mi/Lget_fpreg.c
|
src/mi/Lget_fpreg.c
|
||||||
src/mi/Lset_fpreg.c
|
src/mi/Lset_fpreg.c
|
||||||
src/mi/Lset_caching_policy.c
|
src/mi/Lset_caching_policy.c
|
||||||
|
src/dwarf/Lexpr.c
|
||||||
|
src/dwarf/Lfde.c
|
||||||
|
src/dwarf/Lfind_proc_info-lsb.c
|
||||||
|
src/dwarf/Lparser.c
|
||||||
|
src/dwarf/Lpe.c
|
||||||
|
src/dwarf/global.c
|
||||||
|
src/elf64.c
|
||||||
|
src/os-linux.c
|
||||||
|
|
||||||
|
src/ppc64/is_fpreg.c
|
||||||
|
src/ppc64/regname.c
|
||||||
|
src/ppc64/get_func_addr.c
|
||||||
|
src/ppc/Linit_local.c
|
||||||
|
src/ppc/Linit_remote.c
|
||||||
|
src/ppc/Lis_signal_frame.c
|
||||||
|
src/ppc/longjmp.S
|
||||||
|
src/ppc/Lreg_states_iterate.c
|
||||||
|
src/ppc/siglongjmp.S
|
||||||
|
src/ppc64/setcontext.S
|
||||||
|
src/ppc64/Lcreate_addr_space.c
|
||||||
|
src/ppc64/Lglobal.c
|
||||||
|
src/ppc64/Linit.c
|
||||||
|
src/ppc64/Lreg_states_iterate.c
|
||||||
|
src/ppc64/Lregs.c
|
||||||
|
src/ppc64/Lresume.c
|
||||||
|
src/ppc64/Lstep.c
|
||||||
|
src/ppc64/regname.c
|
||||||
|
src/ppc64/setcontext.S
|
||||||
|
)
|
||||||
|
else ()
|
||||||
|
add_library(unwind
|
||||||
|
src/mi/init.c
|
||||||
|
src/mi/flush_cache.c
|
||||||
|
src/mi/mempool.c
|
||||||
|
src/mi/strerror.c
|
||||||
|
src/mi/_ReadULEB.c
|
||||||
|
src/mi/_ReadSLEB.c
|
||||||
|
src/mi/backtrace.c
|
||||||
|
src/mi/dyn-cancel.c
|
||||||
|
src/mi/dyn-info-list.c
|
||||||
|
src/mi/dyn-register.c
|
||||||
|
src/mi/Ldyn-extract.c
|
||||||
|
src/mi/Lfind_dynamic_proc_info.c
|
||||||
|
src/mi/Lget_accessors.c
|
||||||
|
src/mi/Lget_proc_info_by_ip.c
|
||||||
|
src/mi/Lget_proc_name.c
|
||||||
|
src/mi/Lput_dynamic_unwind_info.c
|
||||||
|
src/mi/Ldestroy_addr_space.c
|
||||||
|
src/mi/Lget_reg.c
|
||||||
|
src/mi/Lset_reg.c
|
||||||
|
src/mi/Lget_fpreg.c
|
||||||
|
src/mi/Lset_fpreg.c
|
||||||
|
src/mi/Lset_caching_policy.c
|
||||||
|
src/dwarf/Lexpr.c
|
||||||
|
src/dwarf/Lfde.c
|
||||||
|
src/dwarf/Lfind_proc_info-lsb.c
|
||||||
|
src/dwarf/Lparser.c
|
||||||
|
src/dwarf/Lpe.c
|
||||||
|
src/dwarf/global.c
|
||||||
|
src/elf64.c
|
||||||
|
src/os-linux.c
|
||||||
|
|
||||||
|
src/x86_64/is_fpreg.c
|
||||||
|
src/x86_64/regname.c
|
||||||
src/x86_64/setcontext.S
|
src/x86_64/setcontext.S
|
||||||
src/x86_64/Lcreate_addr_space.c
|
src/x86_64/Lcreate_addr_space.c
|
||||||
src/x86_64/Lget_save_loc.c
|
src/x86_64/Lget_save_loc.c
|
||||||
@ -40,17 +102,9 @@ src/x86_64/Lstash_frame.c
|
|||||||
src/x86_64/Lstep.c
|
src/x86_64/Lstep.c
|
||||||
src/x86_64/Ltrace.c
|
src/x86_64/Ltrace.c
|
||||||
src/x86_64/getcontext.S
|
src/x86_64/getcontext.S
|
||||||
src/dwarf/Lexpr.c
|
|
||||||
src/dwarf/Lfde.c
|
|
||||||
src/dwarf/Lfind_proc_info-lsb.c
|
|
||||||
src/dwarf/Lparser.c
|
|
||||||
src/dwarf/Lpe.c
|
|
||||||
src/dwarf/global.c
|
|
||||||
src/elf64.c
|
|
||||||
|
|
||||||
src/os-linux.c
|
|
||||||
src/x86_64/Los-linux.c
|
src/x86_64/Los-linux.c
|
||||||
)
|
)
|
||||||
|
endif()
|
||||||
|
|
||||||
find_file (HAVE_ATOMIC_OPS_H "atomic_ops.h")
|
find_file (HAVE_ATOMIC_OPS_H "atomic_ops.h")
|
||||||
configure_file (config/config.h.in ${CMAKE_CURRENT_BINARY_DIR}/config/config.h)
|
configure_file (config/config.h.in ${CMAKE_CURRENT_BINARY_DIR}/config/config.h)
|
||||||
|
|||||||
@ -37,6 +37,7 @@ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. */
|
|||||||
|
|
||||||
#include <stdlib.h>
|
#include <stdlib.h>
|
||||||
#include <libunwind.h>
|
#include <libunwind.h>
|
||||||
|
#include <libunwind-ppc64.h>
|
||||||
|
|
||||||
#include "elf64.h"
|
#include "elf64.h"
|
||||||
#include "mempool.h"
|
#include "mempool.h"
|
||||||
|
|||||||
@ -25,7 +25,7 @@ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
|||||||
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||||
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. */
|
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. */
|
||||||
|
|
||||||
#include "unwind_i.h"
|
#include "../ppc64/unwind_i.h"
|
||||||
|
|
||||||
PROTECTED int
|
PROTECTED int
|
||||||
unw_reg_states_iterate (unw_cursor_t *cursor,
|
unw_reg_states_iterate (unw_cursor_t *cursor,
|
||||||
|
|||||||
@ -3,6 +3,10 @@ SET(LIBRARY_DIR ${ClickHouse_SOURCE_DIR}/contrib/lz4/lib)
|
|||||||
add_library (lz4
|
add_library (lz4
|
||||||
${LIBRARY_DIR}/lz4.c
|
${LIBRARY_DIR}/lz4.c
|
||||||
${LIBRARY_DIR}/lz4hc.c
|
${LIBRARY_DIR}/lz4hc.c
|
||||||
|
${LIBRARY_DIR}/lz4frame.c
|
||||||
|
${LIBRARY_DIR}/lz4frame.h
|
||||||
|
${LIBRARY_DIR}/xxhash.c
|
||||||
|
${LIBRARY_DIR}/xxhash.h
|
||||||
|
|
||||||
${LIBRARY_DIR}/lz4.h
|
${LIBRARY_DIR}/lz4.h
|
||||||
${LIBRARY_DIR}/lz4hc.h
|
${LIBRARY_DIR}/lz4hc.h
|
||||||
|
|||||||
@ -2,8 +2,7 @@
|
|||||||
// MurmurHash2 was written by Austin Appleby, and is placed in the public
|
// MurmurHash2 was written by Austin Appleby, and is placed in the public
|
||||||
// domain. The author hereby disclaims copyright to this source code.
|
// domain. The author hereby disclaims copyright to this source code.
|
||||||
|
|
||||||
#ifndef _MURMURHASH2_H_
|
#pragma once
|
||||||
#define _MURMURHASH2_H_
|
|
||||||
|
|
||||||
//-----------------------------------------------------------------------------
|
//-----------------------------------------------------------------------------
|
||||||
// Platform-specific functions and macros
|
// Platform-specific functions and macros
|
||||||
@ -30,6 +29,3 @@ uint64_t MurmurHash64B (const void * key, int len, uint64_t seed);
|
|||||||
uint32_t MurmurHash2A (const void * key, int len, uint32_t seed);
|
uint32_t MurmurHash2A (const void * key, int len, uint32_t seed);
|
||||||
uint32_t MurmurHashNeutral2 (const void * key, int len, uint32_t seed);
|
uint32_t MurmurHashNeutral2 (const void * key, int len, uint32_t seed);
|
||||||
uint32_t MurmurHashAligned2 (const void * key, int len, uint32_t seed);
|
uint32_t MurmurHashAligned2 (const void * key, int len, uint32_t seed);
|
||||||
|
|
||||||
#endif // _MURMURHASH2_H_
|
|
||||||
|
|
||||||
|
|||||||
@ -2,8 +2,7 @@
|
|||||||
// MurmurHash3 was written by Austin Appleby, and is placed in the public
|
// MurmurHash3 was written by Austin Appleby, and is placed in the public
|
||||||
// domain. The author hereby disclaims copyright to this source code.
|
// domain. The author hereby disclaims copyright to this source code.
|
||||||
|
|
||||||
#ifndef _MURMURHASH3_H_
|
#pragma once
|
||||||
#define _MURMURHASH3_H_
|
|
||||||
|
|
||||||
//-----------------------------------------------------------------------------
|
//-----------------------------------------------------------------------------
|
||||||
// Platform-specific functions and macros
|
// Platform-specific functions and macros
|
||||||
@ -33,5 +32,3 @@ void MurmurHash3_x86_128 ( const void * key, int len, uint32_t seed, void * out
|
|||||||
void MurmurHash3_x64_128 ( const void * key, int len, uint32_t seed, void * out );
|
void MurmurHash3_x64_128 ( const void * key, int len, uint32_t seed, void * out );
|
||||||
|
|
||||||
//-----------------------------------------------------------------------------
|
//-----------------------------------------------------------------------------
|
||||||
|
|
||||||
#endif // _MURMURHASH3_H_
|
|
||||||
|
|||||||
2
contrib/pdqsort/README
Normal file
2
contrib/pdqsort/README
Normal file
@ -0,0 +1,2 @@
|
|||||||
|
Source from https://github.com/orlp/pdqsort
|
||||||
|
Mandatory for Clickhouse, not available in OS packages, we can't use it as submodule.
|
||||||
16
contrib/pdqsort/license.txt
Normal file
16
contrib/pdqsort/license.txt
Normal file
@ -0,0 +1,16 @@
|
|||||||
|
Copyright (c) 2015 Orson Peters <orsonpeters@gmail.com>
|
||||||
|
|
||||||
|
This software is provided 'as-is', without any express or implied warranty. In no event will the
|
||||||
|
authors be held liable for any damages arising from the use of this software.
|
||||||
|
|
||||||
|
Permission is granted to anyone to use this software for any purpose, including commercial
|
||||||
|
applications, and to alter it and redistribute it freely, subject to the following restrictions:
|
||||||
|
|
||||||
|
1. The origin of this software must not be misrepresented; you must not claim that you wrote the
|
||||||
|
original software. If you use this software in a product, an acknowledgment in the product
|
||||||
|
documentation would be appreciated but is not required.
|
||||||
|
|
||||||
|
2. Altered source versions must be plainly marked as such, and must not be misrepresented as
|
||||||
|
being the original software.
|
||||||
|
|
||||||
|
3. This notice may not be removed or altered from any source distribution.
|
||||||
544
contrib/pdqsort/pdqsort.h
Normal file
544
contrib/pdqsort/pdqsort.h
Normal file
@ -0,0 +1,544 @@
|
|||||||
|
/*
|
||||||
|
pdqsort.h - Pattern-defeating quicksort.
|
||||||
|
|
||||||
|
Copyright (c) 2015 Orson Peters
|
||||||
|
|
||||||
|
This software is provided 'as-is', without any express or implied warranty. In no event will the
|
||||||
|
authors be held liable for any damages arising from the use of this software.
|
||||||
|
|
||||||
|
Permission is granted to anyone to use this software for any purpose, including commercial
|
||||||
|
applications, and to alter it and redistribute it freely, subject to the following restrictions:
|
||||||
|
|
||||||
|
1. The origin of this software must not be misrepresented; you must not claim that you wrote the
|
||||||
|
original software. If you use this software in a product, an acknowledgment in the product
|
||||||
|
documentation would be appreciated but is not required.
|
||||||
|
|
||||||
|
2. Altered source versions must be plainly marked as such, and must not be misrepresented as
|
||||||
|
being the original software.
|
||||||
|
|
||||||
|
3. This notice may not be removed or altered from any source distribution.
|
||||||
|
*/
|
||||||
|
|
||||||
|
|
||||||
|
#ifndef PDQSORT_H
|
||||||
|
#define PDQSORT_H
|
||||||
|
|
||||||
|
#include <algorithm>
|
||||||
|
#include <cstddef>
|
||||||
|
#include <functional>
|
||||||
|
#include <utility>
|
||||||
|
#include <iterator>
|
||||||
|
|
||||||
|
#if __cplusplus >= 201103L
|
||||||
|
#include <cstdint>
|
||||||
|
#include <type_traits>
|
||||||
|
#define PDQSORT_PREFER_MOVE(x) std::move(x)
|
||||||
|
#else
|
||||||
|
#define PDQSORT_PREFER_MOVE(x) (x)
|
||||||
|
#endif
|
||||||
|
|
||||||
|
|
||||||
|
namespace pdqsort_detail {
|
||||||
|
enum {
|
||||||
|
// Partitions below this size are sorted using insertion sort.
|
||||||
|
insertion_sort_threshold = 24,
|
||||||
|
|
||||||
|
// Partitions above this size use Tukey's ninther to select the pivot.
|
||||||
|
ninther_threshold = 128,
|
||||||
|
|
||||||
|
// When we detect an already sorted partition, attempt an insertion sort that allows this
|
||||||
|
// amount of element moves before giving up.
|
||||||
|
partial_insertion_sort_limit = 8,
|
||||||
|
|
||||||
|
// Must be multiple of 8 due to loop unrolling, and < 256 to fit in unsigned char.
|
||||||
|
block_size = 64,
|
||||||
|
|
||||||
|
// Cacheline size, assumes power of two.
|
||||||
|
cacheline_size = 64
|
||||||
|
|
||||||
|
};
|
||||||
|
|
||||||
|
#if __cplusplus >= 201103L
|
||||||
|
template<class T> struct is_default_compare : std::false_type { };
|
||||||
|
template<class T> struct is_default_compare<std::less<T>> : std::true_type { };
|
||||||
|
template<class T> struct is_default_compare<std::greater<T>> : std::true_type { };
|
||||||
|
#endif
|
||||||
|
|
||||||
|
// Returns floor(log2(n)), assumes n > 0.
|
||||||
|
template<class T>
|
||||||
|
inline int log2(T n) {
|
||||||
|
int log = 0;
|
||||||
|
while (n >>= 1) ++log;
|
||||||
|
return log;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Sorts [begin, end) using insertion sort with the given comparison function.
|
||||||
|
template<class Iter, class Compare>
|
||||||
|
inline void insertion_sort(Iter begin, Iter end, Compare comp) {
|
||||||
|
typedef typename std::iterator_traits<Iter>::value_type T;
|
||||||
|
if (begin == end) return;
|
||||||
|
|
||||||
|
for (Iter cur = begin + 1; cur != end; ++cur) {
|
||||||
|
Iter sift = cur;
|
||||||
|
Iter sift_1 = cur - 1;
|
||||||
|
|
||||||
|
// Compare first so we can avoid 2 moves for an element already positioned correctly.
|
||||||
|
if (comp(*sift, *sift_1)) {
|
||||||
|
T tmp = PDQSORT_PREFER_MOVE(*sift);
|
||||||
|
|
||||||
|
do { *sift-- = PDQSORT_PREFER_MOVE(*sift_1); }
|
||||||
|
while (sift != begin && comp(tmp, *--sift_1));
|
||||||
|
|
||||||
|
*sift = PDQSORT_PREFER_MOVE(tmp);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Sorts [begin, end) using insertion sort with the given comparison function. Assumes
|
||||||
|
// *(begin - 1) is an element smaller than or equal to any element in [begin, end).
|
||||||
|
template<class Iter, class Compare>
|
||||||
|
inline void unguarded_insertion_sort(Iter begin, Iter end, Compare comp) {
|
||||||
|
typedef typename std::iterator_traits<Iter>::value_type T;
|
||||||
|
if (begin == end) return;
|
||||||
|
|
||||||
|
for (Iter cur = begin + 1; cur != end; ++cur) {
|
||||||
|
Iter sift = cur;
|
||||||
|
Iter sift_1 = cur - 1;
|
||||||
|
|
||||||
|
// Compare first so we can avoid 2 moves for an element already positioned correctly.
|
||||||
|
if (comp(*sift, *sift_1)) {
|
||||||
|
T tmp = PDQSORT_PREFER_MOVE(*sift);
|
||||||
|
|
||||||
|
do { *sift-- = PDQSORT_PREFER_MOVE(*sift_1); }
|
||||||
|
while (comp(tmp, *--sift_1));
|
||||||
|
|
||||||
|
*sift = PDQSORT_PREFER_MOVE(tmp);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Attempts to use insertion sort on [begin, end). Will return false if more than
|
||||||
|
// partial_insertion_sort_limit elements were moved, and abort sorting. Otherwise it will
|
||||||
|
// successfully sort and return true.
|
||||||
|
template<class Iter, class Compare>
|
||||||
|
inline bool partial_insertion_sort(Iter begin, Iter end, Compare comp) {
|
||||||
|
typedef typename std::iterator_traits<Iter>::value_type T;
|
||||||
|
if (begin == end) return true;
|
||||||
|
|
||||||
|
int limit = 0;
|
||||||
|
for (Iter cur = begin + 1; cur != end; ++cur) {
|
||||||
|
if (limit > partial_insertion_sort_limit) return false;
|
||||||
|
|
||||||
|
Iter sift = cur;
|
||||||
|
Iter sift_1 = cur - 1;
|
||||||
|
|
||||||
|
// Compare first so we can avoid 2 moves for an element already positioned correctly.
|
||||||
|
if (comp(*sift, *sift_1)) {
|
||||||
|
T tmp = PDQSORT_PREFER_MOVE(*sift);
|
||||||
|
|
||||||
|
do { *sift-- = PDQSORT_PREFER_MOVE(*sift_1); }
|
||||||
|
while (sift != begin && comp(tmp, *--sift_1));
|
||||||
|
|
||||||
|
*sift = PDQSORT_PREFER_MOVE(tmp);
|
||||||
|
limit += cur - sift;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
template<class Iter, class Compare>
|
||||||
|
inline void sort2(Iter a, Iter b, Compare comp) {
|
||||||
|
if (comp(*b, *a)) std::iter_swap(a, b);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Sorts the elements *a, *b and *c using comparison function comp.
|
||||||
|
template<class Iter, class Compare>
|
||||||
|
inline void sort3(Iter a, Iter b, Iter c, Compare comp) {
|
||||||
|
sort2(a, b, comp);
|
||||||
|
sort2(b, c, comp);
|
||||||
|
sort2(a, b, comp);
|
||||||
|
}
|
||||||
|
|
||||||
|
template<class T>
|
||||||
|
inline T* align_cacheline(T* p) {
|
||||||
|
#if defined(UINTPTR_MAX) && __cplusplus >= 201103L
|
||||||
|
std::uintptr_t ip = reinterpret_cast<std::uintptr_t>(p);
|
||||||
|
#else
|
||||||
|
std::size_t ip = reinterpret_cast<std::size_t>(p);
|
||||||
|
#endif
|
||||||
|
ip = (ip + cacheline_size - 1) & -cacheline_size;
|
||||||
|
return reinterpret_cast<T*>(ip);
|
||||||
|
}
|
||||||
|
|
||||||
|
template<class Iter>
|
||||||
|
inline void swap_offsets(Iter first, Iter last,
|
||||||
|
unsigned char* offsets_l, unsigned char* offsets_r,
|
||||||
|
int num, bool use_swaps) {
|
||||||
|
typedef typename std::iterator_traits<Iter>::value_type T;
|
||||||
|
if (use_swaps) {
|
||||||
|
// This case is needed for the descending distribution, where we need
|
||||||
|
// to have proper swapping for pdqsort to remain O(n).
|
||||||
|
for (int i = 0; i < num; ++i) {
|
||||||
|
std::iter_swap(first + offsets_l[i], last - offsets_r[i]);
|
||||||
|
}
|
||||||
|
} else if (num > 0) {
|
||||||
|
Iter l = first + offsets_l[0]; Iter r = last - offsets_r[0];
|
||||||
|
T tmp(PDQSORT_PREFER_MOVE(*l)); *l = PDQSORT_PREFER_MOVE(*r);
|
||||||
|
for (int i = 1; i < num; ++i) {
|
||||||
|
l = first + offsets_l[i]; *r = PDQSORT_PREFER_MOVE(*l);
|
||||||
|
r = last - offsets_r[i]; *l = PDQSORT_PREFER_MOVE(*r);
|
||||||
|
}
|
||||||
|
*r = PDQSORT_PREFER_MOVE(tmp);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Partitions [begin, end) around pivot *begin using comparison function comp. Elements equal
|
||||||
|
// to the pivot are put in the right-hand partition. Returns the position of the pivot after
|
||||||
|
// partitioning and whether the passed sequence already was correctly partitioned. Assumes the
|
||||||
|
// pivot is a median of at least 3 elements and that [begin, end) is at least
|
||||||
|
// insertion_sort_threshold long. Uses branchless partitioning.
|
||||||
|
template<class Iter, class Compare>
|
||||||
|
inline std::pair<Iter, bool> partition_right_branchless(Iter begin, Iter end, Compare comp) {
|
||||||
|
typedef typename std::iterator_traits<Iter>::value_type T;
|
||||||
|
|
||||||
|
// Move pivot into local for speed.
|
||||||
|
T pivot(PDQSORT_PREFER_MOVE(*begin));
|
||||||
|
Iter first = begin;
|
||||||
|
Iter last = end;
|
||||||
|
|
||||||
|
// Find the first element greater than or equal than the pivot (the median of 3 guarantees
|
||||||
|
// this exists).
|
||||||
|
while (comp(*++first, pivot));
|
||||||
|
|
||||||
|
// Find the first element strictly smaller than the pivot. We have to guard this search if
|
||||||
|
// there was no element before *first.
|
||||||
|
if (first - 1 == begin) while (first < last && !comp(*--last, pivot));
|
||||||
|
else while ( !comp(*--last, pivot));
|
||||||
|
|
||||||
|
// If the first pair of elements that should be swapped to partition are the same element,
|
||||||
|
// the passed in sequence already was correctly partitioned.
|
||||||
|
bool already_partitioned = first >= last;
|
||||||
|
if (!already_partitioned) {
|
||||||
|
std::iter_swap(first, last);
|
||||||
|
++first;
|
||||||
|
}
|
||||||
|
|
||||||
|
// The following branchless partitioning is derived from "BlockQuicksort: How Branch
|
||||||
|
// Mispredictions don’t affect Quicksort" by Stefan Edelkamp and Armin Weiss.
|
||||||
|
unsigned char offsets_l_storage[block_size + cacheline_size];
|
||||||
|
unsigned char offsets_r_storage[block_size + cacheline_size];
|
||||||
|
unsigned char* offsets_l = align_cacheline(offsets_l_storage);
|
||||||
|
unsigned char* offsets_r = align_cacheline(offsets_r_storage);
|
||||||
|
int num_l, num_r, start_l, start_r;
|
||||||
|
num_l = num_r = start_l = start_r = 0;
|
||||||
|
|
||||||
|
while (last - first > 2 * block_size) {
|
||||||
|
// Fill up offset blocks with elements that are on the wrong side.
|
||||||
|
if (num_l == 0) {
|
||||||
|
start_l = 0;
|
||||||
|
Iter it = first;
|
||||||
|
for (unsigned char i = 0; i < block_size;) {
|
||||||
|
offsets_l[num_l] = i++; num_l += !comp(*it, pivot); ++it;
|
||||||
|
offsets_l[num_l] = i++; num_l += !comp(*it, pivot); ++it;
|
||||||
|
offsets_l[num_l] = i++; num_l += !comp(*it, pivot); ++it;
|
||||||
|
offsets_l[num_l] = i++; num_l += !comp(*it, pivot); ++it;
|
||||||
|
offsets_l[num_l] = i++; num_l += !comp(*it, pivot); ++it;
|
||||||
|
offsets_l[num_l] = i++; num_l += !comp(*it, pivot); ++it;
|
||||||
|
offsets_l[num_l] = i++; num_l += !comp(*it, pivot); ++it;
|
||||||
|
offsets_l[num_l] = i++; num_l += !comp(*it, pivot); ++it;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (num_r == 0) {
|
||||||
|
start_r = 0;
|
||||||
|
Iter it = last;
|
||||||
|
for (unsigned char i = 0; i < block_size;) {
|
||||||
|
offsets_r[num_r] = ++i; num_r += comp(*--it, pivot);
|
||||||
|
offsets_r[num_r] = ++i; num_r += comp(*--it, pivot);
|
||||||
|
offsets_r[num_r] = ++i; num_r += comp(*--it, pivot);
|
||||||
|
offsets_r[num_r] = ++i; num_r += comp(*--it, pivot);
|
||||||
|
offsets_r[num_r] = ++i; num_r += comp(*--it, pivot);
|
||||||
|
offsets_r[num_r] = ++i; num_r += comp(*--it, pivot);
|
||||||
|
offsets_r[num_r] = ++i; num_r += comp(*--it, pivot);
|
||||||
|
offsets_r[num_r] = ++i; num_r += comp(*--it, pivot);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Swap elements and update block sizes and first/last boundaries.
|
||||||
|
int num = std::min(num_l, num_r);
|
||||||
|
swap_offsets(first, last, offsets_l + start_l, offsets_r + start_r,
|
||||||
|
num, num_l == num_r);
|
||||||
|
num_l -= num; num_r -= num;
|
||||||
|
start_l += num; start_r += num;
|
||||||
|
if (num_l == 0) first += block_size;
|
||||||
|
if (num_r == 0) last -= block_size;
|
||||||
|
}
|
||||||
|
|
||||||
|
int l_size = 0, r_size = 0;
|
||||||
|
int unknown_left = (last - first) - ((num_r || num_l) ? block_size : 0);
|
||||||
|
if (num_r) {
|
||||||
|
// Handle leftover block by assigning the unknown elements to the other block.
|
||||||
|
l_size = unknown_left;
|
||||||
|
r_size = block_size;
|
||||||
|
} else if (num_l) {
|
||||||
|
l_size = block_size;
|
||||||
|
r_size = unknown_left;
|
||||||
|
} else {
|
||||||
|
// No leftover block, split the unknown elements in two blocks.
|
||||||
|
l_size = unknown_left/2;
|
||||||
|
r_size = unknown_left - l_size;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Fill offset buffers if needed.
|
||||||
|
if (unknown_left && !num_l) {
|
||||||
|
start_l = 0;
|
||||||
|
Iter it = first;
|
||||||
|
for (unsigned char i = 0; i < l_size;) {
|
||||||
|
offsets_l[num_l] = i++; num_l += !comp(*it, pivot); ++it;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (unknown_left && !num_r) {
|
||||||
|
start_r = 0;
|
||||||
|
Iter it = last;
|
||||||
|
for (unsigned char i = 0; i < r_size;) {
|
||||||
|
offsets_r[num_r] = ++i; num_r += comp(*--it, pivot);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
int num = std::min(num_l, num_r);
|
||||||
|
swap_offsets(first, last, offsets_l + start_l, offsets_r + start_r, num, num_l == num_r);
|
||||||
|
num_l -= num; num_r -= num;
|
||||||
|
start_l += num; start_r += num;
|
||||||
|
if (num_l == 0) first += l_size;
|
||||||
|
if (num_r == 0) last -= r_size;
|
||||||
|
|
||||||
|
// We have now fully identified [first, last)'s proper position. Swap the last elements.
|
||||||
|
if (num_l) {
|
||||||
|
offsets_l += start_l;
|
||||||
|
while (num_l--) std::iter_swap(first + offsets_l[num_l], --last);
|
||||||
|
first = last;
|
||||||
|
}
|
||||||
|
if (num_r) {
|
||||||
|
offsets_r += start_r;
|
||||||
|
while (num_r--) std::iter_swap(last - offsets_r[num_r], first), ++first;
|
||||||
|
last = first;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Put the pivot in the right place.
|
||||||
|
Iter pivot_pos = first - 1;
|
||||||
|
*begin = PDQSORT_PREFER_MOVE(*pivot_pos);
|
||||||
|
*pivot_pos = PDQSORT_PREFER_MOVE(pivot);
|
||||||
|
|
||||||
|
return std::make_pair(pivot_pos, already_partitioned);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Partitions [begin, end) around pivot *begin using comparison function comp. Elements equal
|
||||||
|
// to the pivot are put in the right-hand partition. Returns the position of the pivot after
|
||||||
|
// partitioning and whether the passed sequence already was correctly partitioned. Assumes the
|
||||||
|
// pivot is a median of at least 3 elements and that [begin, end) is at least
|
||||||
|
// insertion_sort_threshold long.
|
||||||
|
template<class Iter, class Compare>
|
||||||
|
inline std::pair<Iter, bool> partition_right(Iter begin, Iter end, Compare comp) {
|
||||||
|
typedef typename std::iterator_traits<Iter>::value_type T;
|
||||||
|
|
||||||
|
// Move pivot into local for speed.
|
||||||
|
T pivot(PDQSORT_PREFER_MOVE(*begin));
|
||||||
|
|
||||||
|
Iter first = begin;
|
||||||
|
Iter last = end;
|
||||||
|
|
||||||
|
// Find the first element greater than or equal than the pivot (the median of 3 guarantees
|
||||||
|
// this exists).
|
||||||
|
while (comp(*++first, pivot));
|
||||||
|
|
||||||
|
// Find the first element strictly smaller than the pivot. We have to guard this search if
|
||||||
|
// there was no element before *first.
|
||||||
|
if (first - 1 == begin) while (first < last && !comp(*--last, pivot));
|
||||||
|
else while ( !comp(*--last, pivot));
|
||||||
|
|
||||||
|
// If the first pair of elements that should be swapped to partition are the same element,
|
||||||
|
// the passed in sequence already was correctly partitioned.
|
||||||
|
bool already_partitioned = first >= last;
|
||||||
|
|
||||||
|
// Keep swapping pairs of elements that are on the wrong side of the pivot. Previously
|
||||||
|
// swapped pairs guard the searches, which is why the first iteration is special-cased
|
||||||
|
// above.
|
||||||
|
while (first < last) {
|
||||||
|
std::iter_swap(first, last);
|
||||||
|
while (comp(*++first, pivot));
|
||||||
|
while (!comp(*--last, pivot));
|
||||||
|
}
|
||||||
|
|
||||||
|
// Put the pivot in the right place.
|
||||||
|
Iter pivot_pos = first - 1;
|
||||||
|
*begin = PDQSORT_PREFER_MOVE(*pivot_pos);
|
||||||
|
*pivot_pos = PDQSORT_PREFER_MOVE(pivot);
|
||||||
|
|
||||||
|
return std::make_pair(pivot_pos, already_partitioned);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Similar function to the one above, except elements equal to the pivot are put to the left of
|
||||||
|
// the pivot and it doesn't check or return if the passed sequence already was partitioned.
|
||||||
|
// Since this is rarely used (the many equal case), and in that case pdqsort already has O(n)
|
||||||
|
// performance, no block quicksort is applied here for simplicity.
|
||||||
|
template<class Iter, class Compare>
|
||||||
|
inline Iter partition_left(Iter begin, Iter end, Compare comp) {
|
||||||
|
typedef typename std::iterator_traits<Iter>::value_type T;
|
||||||
|
|
||||||
|
T pivot(PDQSORT_PREFER_MOVE(*begin));
|
||||||
|
Iter first = begin;
|
||||||
|
Iter last = end;
|
||||||
|
|
||||||
|
while (comp(pivot, *--last));
|
||||||
|
|
||||||
|
if (last + 1 == end) while (first < last && !comp(pivot, *++first));
|
||||||
|
else while ( !comp(pivot, *++first));
|
||||||
|
|
||||||
|
while (first < last) {
|
||||||
|
std::iter_swap(first, last);
|
||||||
|
while (comp(pivot, *--last));
|
||||||
|
while (!comp(pivot, *++first));
|
||||||
|
}
|
||||||
|
|
||||||
|
Iter pivot_pos = last;
|
||||||
|
*begin = PDQSORT_PREFER_MOVE(*pivot_pos);
|
||||||
|
*pivot_pos = PDQSORT_PREFER_MOVE(pivot);
|
||||||
|
|
||||||
|
return pivot_pos;
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
template<class Iter, class Compare, bool Branchless>
|
||||||
|
inline void pdqsort_loop(Iter begin, Iter end, Compare comp, int bad_allowed, bool leftmost = true) {
|
||||||
|
typedef typename std::iterator_traits<Iter>::difference_type diff_t;
|
||||||
|
|
||||||
|
// Use a while loop for tail recursion elimination.
|
||||||
|
while (true) {
|
||||||
|
diff_t size = end - begin;
|
||||||
|
|
||||||
|
// Insertion sort is faster for small arrays.
|
||||||
|
if (size < insertion_sort_threshold) {
|
||||||
|
if (leftmost) insertion_sort(begin, end, comp);
|
||||||
|
else unguarded_insertion_sort(begin, end, comp);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Choose pivot as median of 3 or pseudomedian of 9.
|
||||||
|
diff_t s2 = size / 2;
|
||||||
|
if (size > ninther_threshold) {
|
||||||
|
sort3(begin, begin + s2, end - 1, comp);
|
||||||
|
sort3(begin + 1, begin + (s2 - 1), end - 2, comp);
|
||||||
|
sort3(begin + 2, begin + (s2 + 1), end - 3, comp);
|
||||||
|
sort3(begin + (s2 - 1), begin + s2, begin + (s2 + 1), comp);
|
||||||
|
std::iter_swap(begin, begin + s2);
|
||||||
|
} else sort3(begin + s2, begin, end - 1, comp);
|
||||||
|
|
||||||
|
// If *(begin - 1) is the end of the right partition of a previous partition operation
|
||||||
|
// there is no element in [begin, end) that is smaller than *(begin - 1). Then if our
|
||||||
|
// pivot compares equal to *(begin - 1) we change strategy, putting equal elements in
|
||||||
|
// the left partition, greater elements in the right partition. We do not have to
|
||||||
|
// recurse on the left partition, since it's sorted (all equal).
|
||||||
|
if (!leftmost && !comp(*(begin - 1), *begin)) {
|
||||||
|
begin = partition_left(begin, end, comp) + 1;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Partition and get results.
|
||||||
|
std::pair<Iter, bool> part_result =
|
||||||
|
Branchless ? partition_right_branchless(begin, end, comp)
|
||||||
|
: partition_right(begin, end, comp);
|
||||||
|
Iter pivot_pos = part_result.first;
|
||||||
|
bool already_partitioned = part_result.second;
|
||||||
|
|
||||||
|
// Check for a highly unbalanced partition.
|
||||||
|
diff_t l_size = pivot_pos - begin;
|
||||||
|
diff_t r_size = end - (pivot_pos + 1);
|
||||||
|
bool highly_unbalanced = l_size < size / 8 || r_size < size / 8;
|
||||||
|
|
||||||
|
// If we got a highly unbalanced partition we shuffle elements to break many patterns.
|
||||||
|
if (highly_unbalanced) {
|
||||||
|
// If we had too many bad partitions, switch to heapsort to guarantee O(n log n).
|
||||||
|
if (--bad_allowed == 0) {
|
||||||
|
std::make_heap(begin, end, comp);
|
||||||
|
std::sort_heap(begin, end, comp);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (l_size >= insertion_sort_threshold) {
|
||||||
|
std::iter_swap(begin, begin + l_size / 4);
|
||||||
|
std::iter_swap(pivot_pos - 1, pivot_pos - l_size / 4);
|
||||||
|
|
||||||
|
if (l_size > ninther_threshold) {
|
||||||
|
std::iter_swap(begin + 1, begin + (l_size / 4 + 1));
|
||||||
|
std::iter_swap(begin + 2, begin + (l_size / 4 + 2));
|
||||||
|
std::iter_swap(pivot_pos - 2, pivot_pos - (l_size / 4 + 1));
|
||||||
|
std::iter_swap(pivot_pos - 3, pivot_pos - (l_size / 4 + 2));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (r_size >= insertion_sort_threshold) {
|
||||||
|
std::iter_swap(pivot_pos + 1, pivot_pos + (1 + r_size / 4));
|
||||||
|
std::iter_swap(end - 1, end - r_size / 4);
|
||||||
|
|
||||||
|
if (r_size > ninther_threshold) {
|
||||||
|
std::iter_swap(pivot_pos + 2, pivot_pos + (2 + r_size / 4));
|
||||||
|
std::iter_swap(pivot_pos + 3, pivot_pos + (3 + r_size / 4));
|
||||||
|
std::iter_swap(end - 2, end - (1 + r_size / 4));
|
||||||
|
std::iter_swap(end - 3, end - (2 + r_size / 4));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
// If we were decently balanced and we tried to sort an already partitioned
|
||||||
|
// sequence try to use insertion sort.
|
||||||
|
if (already_partitioned && partial_insertion_sort(begin, pivot_pos, comp)
|
||||||
|
&& partial_insertion_sort(pivot_pos + 1, end, comp)) return;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Sort the left partition first using recursion and do tail recursion elimination for
|
||||||
|
// the right-hand partition.
|
||||||
|
pdqsort_loop<Iter, Compare, Branchless>(begin, pivot_pos, comp, bad_allowed, leftmost);
|
||||||
|
begin = pivot_pos + 1;
|
||||||
|
leftmost = false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
template<class Iter, class Compare>
|
||||||
|
inline void pdqsort(Iter begin, Iter end, Compare comp) {
|
||||||
|
if (begin == end) return;
|
||||||
|
|
||||||
|
#if __cplusplus >= 201103L
|
||||||
|
pdqsort_detail::pdqsort_loop<Iter, Compare,
|
||||||
|
pdqsort_detail::is_default_compare<typename std::decay<Compare>::type>::value &&
|
||||||
|
std::is_arithmetic<typename std::iterator_traits<Iter>::value_type>::value>(
|
||||||
|
begin, end, comp, pdqsort_detail::log2(end - begin));
|
||||||
|
#else
|
||||||
|
pdqsort_detail::pdqsort_loop<Iter, Compare, false>(
|
||||||
|
begin, end, comp, pdqsort_detail::log2(end - begin));
|
||||||
|
#endif
|
||||||
|
}
|
||||||
|
|
||||||
|
template<class Iter>
|
||||||
|
inline void pdqsort(Iter begin, Iter end) {
|
||||||
|
typedef typename std::iterator_traits<Iter>::value_type T;
|
||||||
|
pdqsort(begin, end, std::less<T>());
|
||||||
|
}
|
||||||
|
|
||||||
|
template<class Iter, class Compare>
|
||||||
|
inline void pdqsort_branchless(Iter begin, Iter end, Compare comp) {
|
||||||
|
if (begin == end) return;
|
||||||
|
pdqsort_detail::pdqsort_loop<Iter, Compare, true>(
|
||||||
|
begin, end, comp, pdqsort_detail::log2(end - begin));
|
||||||
|
}
|
||||||
|
|
||||||
|
template<class Iter>
|
||||||
|
inline void pdqsort_branchless(Iter begin, Iter end) {
|
||||||
|
typedef typename std::iterator_traits<Iter>::value_type T;
|
||||||
|
pdqsort_branchless(begin, end, std::less<T>());
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
#undef PDQSORT_PREFER_MOVE
|
||||||
|
|
||||||
|
#endif
|
||||||
119
contrib/pdqsort/readme.md
Normal file
119
contrib/pdqsort/readme.md
Normal file
@ -0,0 +1,119 @@
|
|||||||
|
pdqsort
|
||||||
|
-------
|
||||||
|
|
||||||
|
Pattern-defeating quicksort (pdqsort) is a novel sorting algorithm that combines the fast average
|
||||||
|
case of randomized quicksort with the fast worst case of heapsort, while achieving linear time on
|
||||||
|
inputs with certain patterns. pdqsort is an extension and improvement of David Mussers introsort.
|
||||||
|
All code is available for free under the zlib license.
|
||||||
|
|
||||||
|
Best Average Worst Memory Stable Deterministic
|
||||||
|
n n log n n log n log n No Yes
|
||||||
|
|
||||||
|
### Usage
|
||||||
|
|
||||||
|
`pdqsort` is a drop-in replacement for [`std::sort`](http://en.cppreference.com/w/cpp/algorithm/sort).
|
||||||
|
Just replace a call to `std::sort` with `pdqsort` to start using pattern-defeating quicksort. If your
|
||||||
|
comparison function is branchless, you can call `pdqsort_branchless` for a potential big speedup. If
|
||||||
|
you are using C++11, the type you're sorting is arithmetic and your comparison function is not given
|
||||||
|
or is `std::less`/`std::greater`, `pdqsort` automatically delegates to `pdqsort_branchless`.
|
||||||
|
|
||||||
|
### Benchmark
|
||||||
|
|
||||||
|
A comparison of pdqsort and GCC's `std::sort` and `std::stable_sort` with various input
|
||||||
|
distributions:
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Compiled with `-std=c++11 -O2 -m64 -march=native`.
|
||||||
|
|
||||||
|
|
||||||
|
### Visualization
|
||||||
|
|
||||||
|
A visualization of pattern-defeating quicksort sorting a ~200 element array with some duplicates.
|
||||||
|
Generated using Timo Bingmann's [The Sound of Sorting](http://panthema.net/2013/sound-of-sorting/)
|
||||||
|
program, a tool that has been invaluable during the development of pdqsort. For the purposes of
|
||||||
|
this visualization the cutoff point for insertion sort was lowered to 8 elements.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
### The best case
|
||||||
|
|
||||||
|
pdqsort is designed to run in linear time for a couple of best-case patterns. Linear time is
|
||||||
|
achieved for inputs that are in strictly ascending or descending order, only contain equal elements,
|
||||||
|
or are strictly in ascending order followed by one out-of-place element. There are two separate
|
||||||
|
mechanisms at play to achieve this.
|
||||||
|
|
||||||
|
For equal elements a smart partitioning scheme is used that always puts equal elements in the
|
||||||
|
partition containing elements greater than the pivot. When a new pivot is chosen it's compared to
|
||||||
|
the greatest element in the partition before it. If they compare equal we can derive that there are
|
||||||
|
no elements smaller than the chosen pivot. When this happens we switch strategy for this partition,
|
||||||
|
and filter out all elements equal to the pivot.
|
||||||
|
|
||||||
|
To get linear time for the other patterns we check after every partition if any swaps were made. If
|
||||||
|
no swaps were made and the partition was decently balanced we will optimistically attempt to use
|
||||||
|
insertion sort. This insertion sort aborts if more than a constant amount of moves are required to
|
||||||
|
sort.
|
||||||
|
|
||||||
|
|
||||||
|
### The average case
|
||||||
|
|
||||||
|
On average case data where no patterns are detected pdqsort is effectively a quicksort that uses
|
||||||
|
median-of-3 pivot selection, switching to insertion sort if the number of elements to be
|
||||||
|
(recursively) sorted is small. The overhead associated with detecting the patterns for the best case
|
||||||
|
is so small it lies within the error of measurement.
|
||||||
|
|
||||||
|
pdqsort gets a great speedup over the traditional way of implementing quicksort when sorting large
|
||||||
|
arrays (1000+ elements). This is due to a new technique described in "BlockQuicksort: How Branch
|
||||||
|
Mispredictions don't affect Quicksort" by Stefan Edelkamp and Armin Weiss. In short, we bypass the
|
||||||
|
branch predictor by using small buffers (entirely in L1 cache) of the indices of elements that need
|
||||||
|
to be swapped. We fill these buffers in a branch-free way that's quite elegant (in pseudocode):
|
||||||
|
|
||||||
|
```cpp
|
||||||
|
buffer_num = 0; buffer_max_size = 64;
|
||||||
|
for (int i = 0; i < buffer_max_size; ++i) {
|
||||||
|
// With branch:
|
||||||
|
if (elements[i] < pivot) { buffer[buffer_num] = i; buffer_num++; }
|
||||||
|
// Without:
|
||||||
|
buffer[buffer_num] = i; buffer_num += (elements[i] < pivot);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
This is only a speedup if the comparison function itself is branchless, however. By default pdqsort
|
||||||
|
will detect this if you're using C++11 or higher, the type you're sorting is arithmetic (e.g.
|
||||||
|
`int`), and you're using either `std::less` or `std::greater`. You can explicitly request branchless
|
||||||
|
partitioning by calling `pdqsort_branchless` instead of `pdqsort`.
|
||||||
|
|
||||||
|
|
||||||
|
### The worst case
|
||||||
|
|
||||||
|
Quicksort naturally performs bad on inputs that form patterns, due to it being a partition-based
|
||||||
|
sort. Choosing a bad pivot will result in many comparisons that give little to no progress in the
|
||||||
|
sorting process. If the pattern does not get broken up, this can happen many times in a row. Worse,
|
||||||
|
real world data is filled with these patterns.
|
||||||
|
|
||||||
|
Traditionally the solution to this is to randomize the pivot selection of quicksort. While this
|
||||||
|
technically still allows for a quadratic worst case, the chances of it happening are astronomically
|
||||||
|
small. Later, in introsort, pivot selection is kept deterministic, instead switching to the
|
||||||
|
guaranteed O(n log n) heapsort if the recursion depth becomes too big. In pdqsort we adopt a hybrid
|
||||||
|
approach, (deterministically) shuffling some elements to break up patterns when we encounter a "bad"
|
||||||
|
partition. If we encounter too many "bad" partitions we switch to heapsort.
|
||||||
|
|
||||||
|
|
||||||
|
### Bad partitions
|
||||||
|
|
||||||
|
A bad partition occurs when the position of the pivot after partitioning is under 12.5% (1/8th)
|
||||||
|
percentile or over 87,5% percentile - the partition is highly unbalanced. When this happens we will
|
||||||
|
shuffle four elements at fixed locations for both partitions. This effectively breaks up many
|
||||||
|
patterns. If we encounter more than log(n) bad partitions we will switch to heapsort.
|
||||||
|
|
||||||
|
The 1/8th percentile is not chosen arbitrarily. An upper bound of quicksorts worst case runtime can
|
||||||
|
be approximated within a constant factor by the following recurrence:
|
||||||
|
|
||||||
|
T(n, p) = n + T(p(n-1), p) + T((1-p)(n-1), p)
|
||||||
|
|
||||||
|
Where n is the number of elements, and p is the percentile of the pivot after partitioning.
|
||||||
|
`T(n, 1/2)` is the best case for quicksort. On modern systems heapsort is profiled to be
|
||||||
|
approximately 1.8 to 2 times as slow as quicksort. Choosing p such that `T(n, 1/2) / T(n, p) ~= 1.9`
|
||||||
|
as n gets big will ensure that we will only switch to heapsort if it would speed up the sorting.
|
||||||
|
p = 1/8 is a reasonably close value and is cheap to compute on every platform using a bitshift.
|
||||||
1
contrib/snappy
vendored
Submodule
1
contrib/snappy
vendored
Submodule
@ -0,0 +1 @@
|
|||||||
|
Subproject commit 3f194acb57e0487531c96b97af61dcbd025a78a3
|
||||||
2
contrib/ssl
vendored
2
contrib/ssl
vendored
@ -1 +1 @@
|
|||||||
Subproject commit dbbbcdbbd17785566f8f9c107b714f9e213d7293
|
Subproject commit ba8de796195ff9d8bb0249ce289b83226b848b77
|
||||||
1
contrib/thrift
vendored
Submodule
1
contrib/thrift
vendored
Submodule
@ -0,0 +1 @@
|
|||||||
|
Subproject commit 010ccf0a0c7023fea0f6bf4e4078ebdff7e61982
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue
Block a user