diff --git a/.gitmodules b/.gitmodules
index 6ad948c9a0a..6b6b734989d 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -76,3 +76,6 @@
[submodule "contrib/brotli"]
path = contrib/brotli
url = https://github.com/google/brotli.git
+[submodule "contrib/hyperscan"]
+ path = contrib/hyperscan
+ url = https://github.com/ClickHouse-Extras/hyperscan.git
diff --git a/CHANGELOG.md b/CHANGELOG.md
index c29255e6026..7fa13dacdf9 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,3 +1,217 @@
+## ClickHouse release 19.5.2.6, 2019-04-15
+
+### New Features
+
+* [Hyperscan](https://github.com/intel/hyperscan) multiple regular expression matching was added (functions `multiMatchAny`, `multiMatchAnyIndex`, `multiFuzzyMatchAny`, `multiFuzzyMatchAnyIndex`). [#4780](https://github.com/yandex/ClickHouse/pull/4780), [#4841](https://github.com/yandex/ClickHouse/pull/4841) ([Danila Kutenin](https://github.com/danlark1))
+* `multiSearchFirstPosition` function was added. [#4780](https://github.com/yandex/ClickHouse/pull/4780) ([Danila Kutenin](https://github.com/danlark1))
+* Implement the predefined expression filter per row for tables. [#4792](https://github.com/yandex/ClickHouse/pull/4792) ([Ivan](https://github.com/abyss7))
+* A new type of data skipping indices based on bloom filters (can be used for `equal`, `in` and `like` functions). [#4499](https://github.com/yandex/ClickHouse/pull/4499) ([Nikita Vasilev](https://github.com/nikvas0))
+* Added `ASOF JOIN` which allows to run queries that join to the most recent value known. [#4774](https://github.com/yandex/ClickHouse/pull/4774) [#4867](https://github.com/yandex/ClickHouse/pull/4867) [#4863](https://github.com/yandex/ClickHouse/pull/4863) [#4875](https://github.com/yandex/ClickHouse/pull/4875) ([Martijn Bakker](https://github.com/Gladdy), [Artem Zuikov](https://github.com/4ertus2))
+* Rewrite multiple `COMMA JOIN` to `CROSS JOIN`. Then rewrite them to `INNER JOIN` if possible. [#4661](https://github.com/yandex/ClickHouse/pull/4661) ([Artem Zuikov](https://github.com/4ertus2))
+
+### Improvement
+
+* `topK` and `topKWeighted` now supports custom `loadFactor` (fixes issue [#4252](https://github.com/yandex/ClickHouse/issues/4252)). [#4634](https://github.com/yandex/ClickHouse/pull/4634) ([Kirill Danshin](https://github.com/kirillDanshin))
+* Allow to use `parallel_replicas_count > 1` even for tables without sampling (the setting is simply ignored for them). In previous versions it was lead to exception. [#4637](https://github.com/yandex/ClickHouse/pull/4637) ([Alexey Elymanov](https://github.com/digitalist))
+* Support for `CREATE OR REPLACE VIEW`. Allow to create a view or set a new definition in a single statement. [#4654](https://github.com/yandex/ClickHouse/pull/4654) ([Boris Granveaud](https://github.com/bgranvea))
+* `Buffer` table engine now supports `PREWHERE`. [#4671](https://github.com/yandex/ClickHouse/pull/4671) ([Yangkuan Liu](https://github.com/LiuYangkuan))
+* Add ability to start replicated table without metadata in zookeeper in `readonly` mode. [#4691](https://github.com/yandex/ClickHouse/pull/4691) ([alesapin](https://github.com/alesapin))
+* Fixed flicker of progress bar in clickhouse-client. The issue was most noticeable when using `FORMAT Null` with streaming queries. [#4811](https://github.com/yandex/ClickHouse/pull/4811) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Allow to disable functions with `hyperscan` library on per user basis to limit potentially excessive and uncontrolled resource usage. [#4816](https://github.com/yandex/ClickHouse/pull/4816) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Add version number logging in all errors. [#4824](https://github.com/yandex/ClickHouse/pull/4824) ([proller](https://github.com/proller))
+* Added restriction to the `multiMatch` functions which requires string size to fit into `unsigned int`. Also added the number of arguments limit to the `multiSearch` functions. [#4834](https://github.com/yandex/ClickHouse/pull/4834) ([Danila Kutenin](https://github.com/danlark1))
+* Improved usage of scratch space and error handling in Hyperscan. [#4866](https://github.com/yandex/ClickHouse/pull/4866) ([Danila Kutenin](https://github.com/danlark1))
+* Fill `system.graphite_detentions` from a table config of `*GraphiteMergeTree` engine tables. [#4584](https://github.com/yandex/ClickHouse/pull/4584) ([Mikhail f. Shiryaev](https://github.com/Felixoid))
+* Rename `trigramDistance` function to `ngramDistance` and add more functions with `CaseInsensitive` and `UTF`. [#4602](https://github.com/yandex/ClickHouse/pull/4602) ([Danila Kutenin](https://github.com/danlark1))
+* Improved data skipping indices calculation. [#4640](https://github.com/yandex/ClickHouse/pull/4640) ([Nikita Vasilev](https://github.com/nikvas0))
+
+### Bug Fix
+
+* Avoid `std::terminate` in case of memory allocation failure. Now `std::bad_alloc` exception is thrown as expected. [#4665](https://github.com/yandex/ClickHouse/pull/4665) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Fixes capnproto reading from buffer. Sometimes files wasn't loaded successfully by HTTP. [#4674](https://github.com/yandex/ClickHouse/pull/4674) ([Vladislav](https://github.com/smirnov-vs))
+* Fix error `Unknown log entry type: 0` after `OPTIMIZE TABLE FINAL` query. [#4683](https://github.com/yandex/ClickHouse/pull/4683) ([Amos Bird](https://github.com/amosbird))
+* Wrong arguments to `hasAny` or `hasAll` functions may lead to segfault. [#4698](https://github.com/yandex/ClickHouse/pull/4698) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Deadlock may happen while executing `DROP DATABASE dictionary` query. [#4701](https://github.com/yandex/ClickHouse/pull/4701) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Fix undefinied behavior in `median` and `quantile` functions. [#4702](https://github.com/yandex/ClickHouse/pull/4702) ([hcz](https://github.com/hczhcz))
+* Fix compression level detection when `network_compression_method` in lowercase. Broken in v19.1. [#4706](https://github.com/yandex/ClickHouse/pull/4706) ([proller](https://github.com/proller))
+* Keep ordinary, `DEFAULT`, `MATERIALIZED` and `ALIAS` columns in a single list (fixes issue [#2867](https://github.com/yandex/ClickHouse/issues/2867)). [#4707](https://github.com/yandex/ClickHouse/pull/4707) ([Alex Zatelepin](https://github.com/ztlpn))
+* Fixed ignorance of `UTC` setting (fixes issue [#4658](https://github.com/yandex/ClickHouse/issues/4658)). [#4718](https://github.com/yandex/ClickHouse/pull/4718) ([proller](https://github.com/proller))
+* Fix `histogram` function behaviour with `Distributed` tables. [#4741](https://github.com/yandex/ClickHouse/pull/4741) ([olegkv](https://github.com/olegkv))
+* Fixed tsan report `destroy of a locked mutex`. [#4742](https://github.com/yandex/ClickHouse/pull/4742) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Fixed TSan report on shutdown due to race condition in system logs usage. Fixed potential use-after-free on shutdown when part_log is enabled. [#4758](https://github.com/yandex/ClickHouse/pull/4758) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Fix recheck parts in `ReplicatedMergeTreeAlterThread` in case of error. [#4772](https://github.com/yandex/ClickHouse/pull/4772) ([Nikolai Kochetov](https://github.com/KochetovNicolai))
+* Arithmetic operations on intermediate aggregate function states were not working for constant arguments (such as subquery results). [#4776](https://github.com/yandex/ClickHouse/pull/4776) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Always backquote column names in metadata. Otherwise it's impossible to create a table with column named `index` (server won't restart due to malformed `ATTACH` query in metadata). [#4782](https://github.com/yandex/ClickHouse/pull/4782) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Fix crash in `ALTER ... MODIFY ORDER BY` on `Distributed` table. [#4790](https://github.com/yandex/ClickHouse/pull/4790) ([TCeason](https://github.com/TCeason))
+* Fix segfault in `JOIN ON` with enabled `enable_optimize_predicate_expression`. [#4794](https://github.com/yandex/ClickHouse/pull/4794) ([Winter Zhang](https://github.com/zhang2014))
+* Fix bug with adding an extraneous row after consuming a protobuf message from Kafka. [#4808](https://github.com/yandex/ClickHouse/pull/4808) ([Vitaly Baranov](https://github.com/vitlibar))
+* Fix crash of `JOIN` on not-nullable vs nullable column. Fix `NULLs` in right keys in `ANY JOIN` + `join_use_nulls`. [#4815](https://github.com/yandex/ClickHouse/pull/4815) ([Artem Zuikov](https://github.com/4ertus2))
+* Fix segmentation fault in `clickhouse-copier`. [#4835](https://github.com/yandex/ClickHouse/pull/4835) ([proller](https://github.com/proller))
+* Fixed race condition in `SELECT` from `system.tables` if the table is renamed or altered concurrently. [#4836](https://github.com/yandex/ClickHouse/pull/4836) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Fixed data race when fetching data part that is already obsolete. [#4839](https://github.com/yandex/ClickHouse/pull/4839) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Fixed rare data race that can happen during `RENAME` table of MergeTree family. [#4844](https://github.com/yandex/ClickHouse/pull/4844) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Fixed segmentation fault in function `arrayIntersect`. Segmentation fault could happen if function was called with mixed constant and ordinary arguments. [#4847](https://github.com/yandex/ClickHouse/pull/4847) ([Lixiang Qian](https://github.com/fancyqlx))
+* Fixed reading from `Array(LowCardinality)` column in rare case when column contained a long sequence of empty arrays. [#4850](https://github.com/yandex/ClickHouse/pull/4850) ([Nikolai Kochetov](https://github.com/KochetovNicolai))
+* Fix crash in `FULL/RIGHT JOIN` when we joining on nullable vs not nullable. [#4855](https://github.com/yandex/ClickHouse/pull/4855) ([Artem Zuikov](https://github.com/4ertus2))
+* Fix `No message received` exception while fetching parts between replicas. [#4856](https://github.com/yandex/ClickHouse/pull/4856) ([alesapin](https://github.com/alesapin))
+* Fixed `arrayIntersect` function wrong result in case of several repeated values in single array. [#4871](https://github.com/yandex/ClickHouse/pull/4871) ([Nikolai Kochetov](https://github.com/KochetovNicolai))
+* Fix a race condition during concurrent `ALTER COLUMN` queries that could lead to a server crash (fixes issue [#3421](https://github.com/yandex/ClickHouse/issues/3421)). [#4592](https://github.com/yandex/ClickHouse/pull/4592) ([Alex Zatelepin](https://github.com/ztlpn))
+* Fix incorrect result in `FULL/RIGHT JOIN` with const column. [#4723](https://github.com/yandex/ClickHouse/pull/4723) ([Artem Zuikov](https://github.com/4ertus2))
+* Fix duplicates in `GLOBAL JOIN` with asterisk. [#4705](https://github.com/yandex/ClickHouse/pull/4705) ([Artem Zuikov](https://github.com/4ertus2))
+* Fix parameter deduction in `ALTER MODIFY` of column `CODEC` when column type is not specified. [#4883](https://github.com/yandex/ClickHouse/pull/4883) ([alesapin](https://github.com/alesapin))
+* Functions `cutQueryStringAndFragment()` and `queryStringAndFragment()` now works correctly when `URL` contains a fragment and no query. [#4894](https://github.com/yandex/ClickHouse/pull/4894) ([Vitaly Baranov](https://github.com/vitlibar))
+* Fix rare bug when setting `min_bytes_to_use_direct_io` is greater than zero, which occures when thread have to seek backward in column file. [#4897](https://github.com/yandex/ClickHouse/pull/4897) ([alesapin](https://github.com/alesapin))
+* Fix wrong argument types for aggregate functions with `LowCardinality` arguments (fixes issue [#4919](https://github.com/yandex/ClickHouse/issues/4919)). [#4922](https://github.com/yandex/ClickHouse/pull/4922) ([Nikolai Kochetov](https://github.com/KochetovNicolai))
+* Fix wrong name qualification in `GLOBAL JOIN`. [#4969](https://github.com/yandex/ClickHouse/pull/4969) ([Artem Zuikov](https://github.com/4ertus2))
+* Function `toISOWeek` result for year 1970. [#4988](https://github.com/yandex/ClickHouse/pull/4988) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Fix `DROP`, `TRUNCATE` and `OPTIMIZE` queries duplication, when executed on `ON CLUSTER` for `ReplicatedMergeTree*` tables family. [#4991](https://github.com/yandex/ClickHouse/pull/4991) ([alesapin](https://github.com/alesapin))
+
+### Backward Incompatible Change
+
+* Rename setting `insert_sample_with_metadata` to setting `input_format_defaults_for_omitted_fields`. [#4771](https://github.com/yandex/ClickHouse/pull/4771) ([Artem Zuikov](https://github.com/4ertus2))
+* Added setting `max_partitions_per_insert_block` (with value 100 by default). If inserted block contains larger number of partitions, an exception is thrown. Set it to 0 if you want to remove the limit (not recommended). [#4845](https://github.com/yandex/ClickHouse/pull/4845) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Multi-search functions were renamed (`multiPosition` to `multiSearchAllPositions`, `multiSearch` to `multiSearchAny`, `firstMatch` to `multiSearchFirstIndex`). [#4780](https://github.com/yandex/ClickHouse/pull/4780) ([Danila Kutenin](https://github.com/danlark1))
+
+### Performance Improvement
+
+* Optimize Volnitsky searcher by inlining, giving about 5-10% search improvement for queries with many needles or many similar bigrams. [#4862](https://github.com/yandex/ClickHouse/pull/4862) ([Danila Kutenin](https://github.com/danlark1))
+* Fix performance issue when setting `use_uncompressed_cache` is greater than zero, which appeared when all read data contained in cache. [#4913](https://github.com/yandex/ClickHouse/pull/4913) ([alesapin](https://github.com/alesapin))
+
+
+### Build/Testing/Packaging Improvement
+
+* Hardening debug build: more granular memory mappings and ASLR; add memory protection for mark cache and index. This allows to find more memory stomping bugs in case when ASan and MSan cannot do it. [#4632](https://github.com/yandex/ClickHouse/pull/4632) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Add support for cmake variables `ENABLE_PROTOBUF`, `ENABLE_PARQUET` and `ENABLE_BROTLI` which allows to enable/disable the above features (same as we can do for librdkafka, mysql, etc). [#4669](https://github.com/yandex/ClickHouse/pull/4669) ([Silviu Caragea](https://github.com/silviucpp))
+* Add ability to print process list and stacktraces of all threads if some queries are hung after test run. [#4675](https://github.com/yandex/ClickHouse/pull/4675) ([alesapin](https://github.com/alesapin))
+* Add retries on `Connection loss` error in `clickhouse-test`. [#4682](https://github.com/yandex/ClickHouse/pull/4682) ([alesapin](https://github.com/alesapin))
+* Add freebsd build with vagrant and build with thread sanitizer to packager script. [#4712](https://github.com/yandex/ClickHouse/pull/4712) [#4748](https://github.com/yandex/ClickHouse/pull/4748) ([alesapin](https://github.com/alesapin))
+* Now user asked for password for user `'default'` during installation. [#4725](https://github.com/yandex/ClickHouse/pull/4725) ([proller](https://github.com/proller))
+* Suppress warning in `rdkafka` library. [#4740](https://github.com/yandex/ClickHouse/pull/4740) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Allow ability to build without ssl. [#4750](https://github.com/yandex/ClickHouse/pull/4750) ([proller](https://github.com/proller))
+* Add a way to launch clickhouse-server image from a custom user. [#4753](https://github.com/yandex/ClickHouse/pull/4753) ([Mikhail f. Shiryaev](https://github.com/Felixoid))
+* Upgrade contrib boost to 1.69. [#4793](https://github.com/yandex/ClickHouse/pull/4793) ([proller](https://github.com/proller))
+* Disable usage of `mremap` when compiled with Thread Sanitizer. Surprisingly enough, TSan does not intercept `mremap` (though it does intercept `mmap`, `munmap`) that leads to false positives. Fixed TSan report in stateful tests. [#4859](https://github.com/yandex/ClickHouse/pull/4859) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Add test checking using format schema via HTTP interface. [#4864](https://github.com/yandex/ClickHouse/pull/4864) ([Vitaly Baranov](https://github.com/vitlibar))
+
+## ClickHouse release 19.4.3.11, 2019-04-02
+
+### Bug Fixes
+
+* Fix crash in `FULL/RIGHT JOIN` when we joining on nullable vs not nullable. [#4855](https://github.com/yandex/ClickHouse/pull/4855) ([Artem Zuikov](https://github.com/4ertus2))
+* Fix segmentation fault in `clickhouse-copier`. [#4835](https://github.com/yandex/ClickHouse/pull/4835) ([proller](https://github.com/proller))
+
+### Build/Testing/Packaging Improvement
+
+* Add a way to launch clickhouse-server image from a custom user. [#4753](https://github.com/yandex/ClickHouse/pull/4753) ([Mikhail f. Shiryaev](https://github.com/Felixoid))
+
+## ClickHouse release 19.4.2.7, 2019-03-30
+
+### Bug Fixes
+* Fixed reading from `Array(LowCardinality)` column in rare case when column contained a long sequence of empty arrays. [#4850](https://github.com/yandex/ClickHouse/pull/4850) ([Nikolai Kochetov](https://github.com/KochetovNicolai))
+
+## ClickHouse release 19.4.1.3, 2019-03-19
+
+### Bug Fixes
+* Fixed remote queries which contain both `LIMIT BY` and `LIMIT`. Previously, if `LIMIT BY` and `LIMIT` were used for remote query, `LIMIT` could happen before `LIMIT BY`, which led to too filtered result. [#4708](https://github.com/yandex/ClickHouse/pull/4708) ([Constantin S. Pan](https://github.com/kvap))
+
+## ClickHouse release 19.4.0.49, 2019-03-09
+
+### New Features
+* Added full support for `Protobuf` format (input and output, nested data structures). [#4174](https://github.com/yandex/ClickHouse/pull/4174) [#4493](https://github.com/yandex/ClickHouse/pull/4493) ([Vitaly Baranov](https://github.com/vitlibar))
+* Added bitmap functions with Roaring Bitmaps. [#4207](https://github.com/yandex/ClickHouse/pull/4207) ([Andy Yang](https://github.com/andyyzh)) [#4568](https://github.com/yandex/ClickHouse/pull/4568) ([Vitaly Baranov](https://github.com/vitlibar))
+* Parquet format support. [#4448](https://github.com/yandex/ClickHouse/pull/4448) ([proller](https://github.com/proller))
+* N-gram distance was added for fuzzy string comparison. It is similar to q-gram metrics in R language. [#4466](https://github.com/yandex/ClickHouse/pull/4466) ([Danila Kutenin](https://github.com/danlark1))
+* Combine rules for graphite rollup from dedicated aggregation and retention patterns. [#4426](https://github.com/yandex/ClickHouse/pull/4426) ([Mikhail f. Shiryaev](https://github.com/Felixoid))
+* Added `max_execution_speed` and `max_execution_speed_bytes` to limit resource usage. Added `min_execution_speed_bytes` setting to complement the `min_execution_speed`. [#4430](https://github.com/yandex/ClickHouse/pull/4430) ([Winter Zhang](https://github.com/zhang2014))
+* Implemented function `flatten`. [#4555](https://github.com/yandex/ClickHouse/pull/4555) [#4409](https://github.com/yandex/ClickHouse/pull/4409) ([alexey-milovidov](https://github.com/alexey-milovidov), [kzon](https://github.com/kzon))
+* Added functions `arrayEnumerateDenseRanked` and `arrayEnumerateUniqRanked` (it's like `arrayEnumerateUniq` but allows to fine tune array depth to look inside multidimensional arrays). [#4475](https://github.com/yandex/ClickHouse/pull/4475) ([proller](https://github.com/proller)) [#4601](https://github.com/yandex/ClickHouse/pull/4601) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Multiple JOINS with some restrictions: no asterisks, no complex aliases in ON/WHERE/GROUP BY/... [#4462](https://github.com/yandex/ClickHouse/pull/4462) ([Artem Zuikov](https://github.com/4ertus2))
+
+### Bug Fixes
+* This release also contains all bug fixes from 19.3 and 19.1.
+* Fixed bug in data skipping indices: order of granules after INSERT was incorrect. [#4407](https://github.com/yandex/ClickHouse/pull/4407) ([Nikita Vasilev](https://github.com/nikvas0))
+* Fixed `set` index for `Nullable` and `LowCardinality` columns. Before it, `set` index with `Nullable` or `LowCardinality` column led to error `Data type must be deserialized with multiple streams` while selecting. [#4594](https://github.com/yandex/ClickHouse/pull/4594) ([Nikolai Kochetov](https://github.com/KochetovNicolai))
+* Correctly set update_time on full `executable` dictionary update. [#4551](https://github.com/yandex/ClickHouse/pull/4551) ([Tema Novikov](https://github.com/temoon))
+* Fix broken progress bar in 19.3. [#4627](https://github.com/yandex/ClickHouse/pull/4627) ([filimonov](https://github.com/filimonov))
+* Fixed inconsistent values of MemoryTracker when memory region was shrinked, in certain cases. [#4619](https://github.com/yandex/ClickHouse/pull/4619) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Fixed undefined behaviour in ThreadPool. [#4612](https://github.com/yandex/ClickHouse/pull/4612) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Fixed a very rare crash with the message `mutex lock failed: Invalid argument` that could happen when a MergeTree table was dropped concurrently with a SELECT. [#4608](https://github.com/yandex/ClickHouse/pull/4608) ([Alex Zatelepin](https://github.com/ztlpn))
+* ODBC driver compatibility with `LowCardinality` data type. [#4381](https://github.com/yandex/ClickHouse/pull/4381) ([proller](https://github.com/proller))
+* FreeBSD: Fixup for `AIOcontextPool: Found io_event with unknown id 0` error. [#4438](https://github.com/yandex/ClickHouse/pull/4438) ([urgordeadbeef](https://github.com/urgordeadbeef))
+* `system.part_log` table was created regardless to configuration. [#4483](https://github.com/yandex/ClickHouse/pull/4483) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Fix undefined behaviour in `dictIsIn` function for cache dictionaries. [#4515](https://github.com/yandex/ClickHouse/pull/4515) ([alesapin](https://github.com/alesapin))
+* Fixed a deadlock when a SELECT query locks the same table multiple times (e.g. from different threads or when executing multiple subqueries) and there is a concurrent DDL query. [#4535](https://github.com/yandex/ClickHouse/pull/4535) ([Alex Zatelepin](https://github.com/ztlpn))
+* Disable compile_expressions by default until we get own `llvm` contrib and can test it with `clang` and `asan`. [#4579](https://github.com/yandex/ClickHouse/pull/4579) ([alesapin](https://github.com/alesapin))
+* Prevent `std::terminate` when `invalidate_query` for `clickhouse` external dictionary source has returned wrong resultset (empty or more than one row or more than one column). Fixed issue when the `invalidate_query` was performed every five seconds regardless to the `lifetime`. [#4583](https://github.com/yandex/ClickHouse/pull/4583) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Avoid deadlock when the `invalidate_query` for a dictionary with `clickhouse` source was involving `system.dictionaries` table or `Dictionaries` database (rare case). [#4599](https://github.com/yandex/ClickHouse/pull/4599) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Fixes for CROSS JOIN with empty WHERE. [#4598](https://github.com/yandex/ClickHouse/pull/4598) ([Artem Zuikov](https://github.com/4ertus2))
+* Fixed segfault in function "replicate" when constant argument is passed. [#4603](https://github.com/yandex/ClickHouse/pull/4603) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Fix lambda function with predicate optimizer. [#4408](https://github.com/yandex/ClickHouse/pull/4408) ([Winter Zhang](https://github.com/zhang2014))
+* Multiple JOINs multiple fixes. [#4595](https://github.com/yandex/ClickHouse/pull/4595) ([Artem Zuikov](https://github.com/4ertus2))
+
+### Improvements
+* Support aliases in JOIN ON section for right table columns. [#4412](https://github.com/yandex/ClickHouse/pull/4412) ([Artem Zuikov](https://github.com/4ertus2))
+* Result of multiple JOINs need correct result names to be used in subselects. Replace flat aliases with source names in result. [#4474](https://github.com/yandex/ClickHouse/pull/4474) ([Artem Zuikov](https://github.com/4ertus2))
+* Improve push-down logic for joined statements. [#4387](https://github.com/yandex/ClickHouse/pull/4387) ([Ivan](https://github.com/abyss7))
+
+### Performance Improvements
+* Improved heuristics of "move to PREWHERE" optimization. [#4405](https://github.com/yandex/ClickHouse/pull/4405) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Use proper lookup tables that uses HashTable's API for 8-bit and 16-bit keys. [#4536](https://github.com/yandex/ClickHouse/pull/4536) ([Amos Bird](https://github.com/amosbird))
+* Improved performance of string comparison. [#4564](https://github.com/yandex/ClickHouse/pull/4564) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Cleanup distributed DDL queue in a separate thread so that it doesn't slow down the main loop that processes distributed DDL tasks. [#4502](https://github.com/yandex/ClickHouse/pull/4502) ([Alex Zatelepin](https://github.com/ztlpn))
+* When `min_bytes_to_use_direct_io` is set to 1, not every file was opened with O_DIRECT mode because the data size to read was sometimes underestimated by the size of one compressed block. [#4526](https://github.com/yandex/ClickHouse/pull/4526) ([alexey-milovidov](https://github.com/alexey-milovidov))
+
+### Build/Testing/Packaging Improvement
+* Added support for clang-9 [#4604](https://github.com/yandex/ClickHouse/pull/4604) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Fix wrong `__asm__` instructions (again) [#4621](https://github.com/yandex/ClickHouse/pull/4621) ([Konstantin Podshumok](https://github.com/podshumok))
+* Add ability to specify settings for `clickhouse-performance-test` from command line. [#4437](https://github.com/yandex/ClickHouse/pull/4437) ([alesapin](https://github.com/alesapin))
+* Add dictionaries tests to integration tests. [#4477](https://github.com/yandex/ClickHouse/pull/4477) ([alesapin](https://github.com/alesapin))
+* Added queries from the benchmark on the website to automated performance tests. [#4496](https://github.com/yandex/ClickHouse/pull/4496) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* `xxhash.h` does not exist in external lz4 because it is an implementation detail and its symbols are namespaced with `XXH_NAMESPACE` macro. When lz4 is external, xxHash has to be external too, and the dependents have to link to it. [#4495](https://github.com/yandex/ClickHouse/pull/4495) ([Orivej Desh](https://github.com/orivej))
+* Fixed a case when `quantileTiming` aggregate function can be called with negative or floating point argument (this fixes fuzz test with undefined behaviour sanitizer). [#4506](https://github.com/yandex/ClickHouse/pull/4506) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Spelling error correction. [#4531](https://github.com/yandex/ClickHouse/pull/4531) ([sdk2](https://github.com/sdk2))
+* Fix compilation on Mac. [#4371](https://github.com/yandex/ClickHouse/pull/4371) ([Vitaly Baranov](https://github.com/vitlibar))
+* Build fixes for FreeBSD and various unusual build configurations. [#4444](https://github.com/yandex/ClickHouse/pull/4444) ([proller](https://github.com/proller))
+
+## ClickHouse release 19.3.9.1, 2019-04-02
+
+### Bug Fixes
+
+* Fix crash in `FULL/RIGHT JOIN` when we joining on nullable vs not nullable. [#4855](https://github.com/yandex/ClickHouse/pull/4855) ([Artem Zuikov](https://github.com/4ertus2))
+* Fix segmentation fault in `clickhouse-copier`. [#4835](https://github.com/yandex/ClickHouse/pull/4835) ([proller](https://github.com/proller))
+* Fixed reading from `Array(LowCardinality)` column in rare case when column contained a long sequence of empty arrays. [#4850](https://github.com/yandex/ClickHouse/pull/4850) ([Nikolai Kochetov](https://github.com/KochetovNicolai))
+
+### Build/Testing/Packaging Improvement
+
+* Add a way to launch clickhouse-server image from a custom user [#4753](https://github.com/yandex/ClickHouse/pull/4753) ([Mikhail f. Shiryaev](https://github.com/Felixoid))
+
+
+## ClickHouse release 19.3.7, 2019-03-12
+
+### Bug fixes
+
+* Fixed error in #3920. This error manifestate itself as random cache corruption (messages `Unknown codec family code`, `Cannot seek through file`) and segfaults. This bug first appeared in version 19.1 and is present in versions up to 19.1.10 and 19.3.6. [#4623](https://github.com/yandex/ClickHouse/pull/4623) ([alexey-milovidov](https://github.com/alexey-milovidov))
+
+
+## ClickHouse release 19.3.6, 2019-03-02
+
+### Bug fixes
+
+* When there are more than 1000 threads in a thread pool, `std::terminate` may happen on thread exit. [Azat Khuzhin](https://github.com/azat) [#4485](https://github.com/yandex/ClickHouse/pull/4485) [#4505](https://github.com/yandex/ClickHouse/pull/4505) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Now it's possible to create `ReplicatedMergeTree*` tables with comments on columns without defaults and tables with columns codecs without comments and defaults. Also fix comparison of codecs. [#4523](https://github.com/yandex/ClickHouse/pull/4523) ([alesapin](https://github.com/alesapin))
+* Fixed crash on JOIN with array or tuple. [#4552](https://github.com/yandex/ClickHouse/pull/4552) ([Artem Zuikov](https://github.com/4ertus2))
+* Fixed crash in clickhouse-copier with the message `ThreadStatus not created`. [#4540](https://github.com/yandex/ClickHouse/pull/4540) ([Artem Zuikov](https://github.com/4ertus2))
+* Fixed hangup on server shutdown if distributed DDLs were used. [#4472](https://github.com/yandex/ClickHouse/pull/4472) ([Alex Zatelepin](https://github.com/ztlpn))
+* Incorrect column numbers were printed in error message about text format parsing for columns with number greater than 10. [#4484](https://github.com/yandex/ClickHouse/pull/4484) ([alexey-milovidov](https://github.com/alexey-milovidov))
+
+### Build/Testing/Packaging Improvements
+
+* Fixed build with AVX enabled. [#4527](https://github.com/yandex/ClickHouse/pull/4527) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Enable extended accounting and IO accounting based on good known version instead of kernel under which it is compiled. [#4541](https://github.com/yandex/ClickHouse/pull/4541) ([nvartolomei](https://github.com/nvartolomei))
+* Allow to skip setting of core_dump.size_limit, warning instead of throw if limit set fail. [#4473](https://github.com/yandex/ClickHouse/pull/4473) ([proller](https://github.com/proller))
+* Removed the `inline` tags of `void readBinary(...)` in `Field.cpp`. Also merged redundant `namespace DB` blocks. [#4530](https://github.com/yandex/ClickHouse/pull/4530) ([hcz](https://github.com/hczhcz))
+
+
## ClickHouse release 19.3.5, 2019-02-21
### Bug fixes
@@ -67,7 +281,7 @@
* Fixed race condition when selecting from `system.tables` may give `table doesn't exist` error. [#4313](https://github.com/yandex/ClickHouse/pull/4313) ([alexey-milovidov](https://github.com/alexey-milovidov))
* `clickhouse-client` can segfault on exit while loading data for command line suggestions if it was run in interactive mode. [#4317](https://github.com/yandex/ClickHouse/pull/4317) ([alexey-milovidov](https://github.com/alexey-milovidov))
* Fixed a bug when the execution of mutations containing `IN` operators was producing incorrect results. [#4099](https://github.com/yandex/ClickHouse/pull/4099) ([Alex Zatelepin](https://github.com/ztlpn))
-* Fixed error: if there is a database with `Dictionary` engine, all dictionaries forced to load at server startup, and if there is a dictionary with ClickHouse source from localhost, the dictionary cannot load. [#4255](https://github.com/yandex/ClickHouse/pull/4255) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Fixed error: if there is a database with `Dictionary` engine, all dictionaries forced to load at server startup, and if there is a dictionary with ClickHouse source from localhost, the dictionary cannot load. [#4255](https://github.com/yandex/ClickHouse/pull/4255) ([alexey-milovidov](https://github.com/alexey-milovidov))
* Fixed error when system logs are tried to create again at server shutdown. [#4254](https://github.com/yandex/ClickHouse/pull/4254) ([alexey-milovidov](https://github.com/alexey-milovidov))
* Correctly return the right type and properly handle locks in `joinGet` function. [#4153](https://github.com/yandex/ClickHouse/pull/4153) ([Amos Bird](https://github.com/amosbird))
* Added `sumMapWithOverflow` function. [#4151](https://github.com/yandex/ClickHouse/pull/4151) ([Léo Ercolanelli](https://github.com/ercolanelli-leo))
@@ -92,7 +306,7 @@
* Added script which creates changelog from pull requests description. [#4169](https://github.com/yandex/ClickHouse/pull/4169) [#4173](https://github.com/yandex/ClickHouse/pull/4173) ([KochetovNicolai](https://github.com/KochetovNicolai)) ([KochetovNicolai](https://github.com/KochetovNicolai))
* Added puppet module for Clickhouse. [#4182](https://github.com/yandex/ClickHouse/pull/4182) ([Maxim Fedotov](https://github.com/MaxFedotov))
* Added docs for a group of undocumented functions. [#4168](https://github.com/yandex/ClickHouse/pull/4168) ([Winter Zhang](https://github.com/zhang2014))
-* ARM build fixes. [#4210](https://github.com/yandex/ClickHouse/pull/4210)[#4306](https://github.com/yandex/ClickHouse/pull/4306) [#4291](https://github.com/yandex/ClickHouse/pull/4291) ([proller](https://github.com/proller)) ([proller](https://github.com/proller))
+* ARM build fixes. [#4210](https://github.com/yandex/ClickHouse/pull/4210)[#4306](https://github.com/yandex/ClickHouse/pull/4306) [#4291](https://github.com/yandex/ClickHouse/pull/4291) ([proller](https://github.com/proller)) ([proller](https://github.com/proller))
* Dictionary tests now able to run from `ctest`. [#4189](https://github.com/yandex/ClickHouse/pull/4189) ([proller](https://github.com/proller))
* Now `/etc/ssl` is used as default directory with SSL certificates. [#4167](https://github.com/yandex/ClickHouse/pull/4167) ([alexey-milovidov](https://github.com/alexey-milovidov))
* Added checking SSE and AVX instruction at start. [#4234](https://github.com/yandex/ClickHouse/pull/4234) ([Igr](https://github.com/igron99))
@@ -123,6 +337,20 @@
* Improved server shutdown time and ALTERs waiting time. [#4372](https://github.com/yandex/ClickHouse/pull/4372) ([alexey-milovidov](https://github.com/alexey-milovidov))
* Added info about the replicated_can_become_leader setting to system.replicas and add logging if the replica won't try to become leader. [#4379](https://github.com/yandex/ClickHouse/pull/4379) ([Alex Zatelepin](https://github.com/ztlpn))
+
+## ClickHouse release 19.1.14, 2019-03-14
+
+* Fixed error `Column ... queried more than once` that may happen if the setting `asterisk_left_columns_only` is set to 1 in case of using `GLOBAL JOIN` with `SELECT *` (rare case). The issue does not exist in 19.3 and newer. [6bac7d8d](https://github.com/yandex/ClickHouse/pull/4692/commits/6bac7d8d11a9b0d6de0b32b53c47eb2f6f8e7062) ([Artem Zuikov](https://github.com/4ertus2))
+
+## ClickHouse release 19.1.13, 2019-03-12
+
+This release contains exactly the same set of patches as 19.3.7.
+
+## ClickHouse release 19.1.10, 2019-03-03
+
+This release contains exactly the same set of patches as 19.3.6.
+
+
## ClickHouse release 19.1.9, 2019-02-21
### Bug fixes
@@ -140,7 +368,7 @@
### Bug Fixes
* Correctly return the right type and properly handle locks in `joinGet` function. [#4153](https://github.com/yandex/ClickHouse/pull/4153) ([Amos Bird](https://github.com/amosbird))
* Fixed error when system logs are tried to create again at server shutdown. [#4254](https://github.com/yandex/ClickHouse/pull/4254) ([alexey-milovidov](https://github.com/alexey-milovidov))
-* Fixed error: if there is a database with `Dictionary` engine, all dictionaries forced to load at server startup, and if there is a dictionary with ClickHouse source from localhost, the dictionary cannot load. [#4255](https://github.com/yandex/ClickHouse/pull/4255) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Fixed error: if there is a database with `Dictionary` engine, all dictionaries forced to load at server startup, and if there is a dictionary with ClickHouse source from localhost, the dictionary cannot load. [#4255](https://github.com/yandex/ClickHouse/pull/4255) ([alexey-milovidov](https://github.com/alexey-milovidov))
* Fixed a bug when the execution of mutations containing `IN` operators was producing incorrect results. [#4099](https://github.com/yandex/ClickHouse/pull/4099) ([Alex Zatelepin](https://github.com/ztlpn))
* `clickhouse-client` can segfault on exit while loading data for command line suggestions if it was run in interactive mode. [#4317](https://github.com/yandex/ClickHouse/pull/4317) ([alexey-milovidov](https://github.com/alexey-milovidov))
* Fixed race condition when selecting from `system.tables` may give `table doesn't exist` error. [#4313](https://github.com/yandex/ClickHouse/pull/4313) ([alexey-milovidov](https://github.com/alexey-milovidov))
diff --git a/CHANGELOG_RU.md b/CHANGELOG_RU.md
index 3c3b425bb7f..743edeb7eeb 100644
--- a/CHANGELOG_RU.md
+++ b/CHANGELOG_RU.md
@@ -1,3 +1,89 @@
+## ClickHouse release 19.4.0.49, 2019-03-09
+
+### Новые возможности
+* Добавлена полная поддержка формата `Protobuf` (чтение и запись, вложенные структуры данных). [#4174](https://github.com/yandex/ClickHouse/pull/4174) [#4493](https://github.com/yandex/ClickHouse/pull/4493) ([Vitaly Baranov](https://github.com/vitlibar))
+* Добавлены функции для работы с битовыми масками с использованием библиотеки Roaring Bitmaps. [#4207](https://github.com/yandex/ClickHouse/pull/4207) ([Andy Yang](https://github.com/andyyzh)) [#4568](https://github.com/yandex/ClickHouse/pull/4568) ([Vitaly Baranov](https://github.com/vitlibar))
+* Поддержка формата `Parquet` [#4448](https://github.com/yandex/ClickHouse/pull/4448) ([proller](https://github.com/proller))
+* Вычисление расстояния между строками с помощью подсчёта N-грам - для приближённого сравнения строк. Алгоритм похож на q-gram metrics в языке R. [#4466](https://github.com/yandex/ClickHouse/pull/4466) ([Danila Kutenin](https://github.com/danlark1))
+* Движок таблиц GraphiteMergeTree поддерживает отдельные шаблоны для правил агрегации и для правил времени хранения. [#4426](https://github.com/yandex/ClickHouse/pull/4426) ([Mikhail f. Shiryaev](https://github.com/Felixoid))
+* Добавлены настройки `max_execution_speed` и `max_execution_speed_bytes` для того, чтобы ограничить потребление ресурсов запросами. Добавлена настройка `min_execution_speed_bytes` в дополнение к `min_execution_speed`. [#4430](https://github.com/yandex/ClickHouse/pull/4430) ([Winter Zhang](https://github.com/zhang2014))
+* Добавлена функция `flatten` - конвертация многомерных массивов в плоский массив. [#4555](https://github.com/yandex/ClickHouse/pull/4555) [#4409](https://github.com/yandex/ClickHouse/pull/4409) ([alexey-milovidov](https://github.com/alexey-milovidov), [kzon](https://github.com/kzon))
+* Добавлены функции `arrayEnumerateDenseRanked` и `arrayEnumerateUniqRanked` (похожа на `arrayEnumerateUniq` но позволяет указать глубину, на которую следует смотреть в многомерные массивы). [#4475](https://github.com/yandex/ClickHouse/pull/4475) ([proller](https://github.com/proller)) [#4601](https://github.com/yandex/ClickHouse/pull/4601) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Добавлена поддержка множества JOIN в одном запросе без подзапросов, с некоторыми ограничениями: без звёздочки и без алиасов сложных выражений в ON/WHERE/GROUP BY/... [#4462](https://github.com/yandex/ClickHouse/pull/4462) ([Artem Zuikov](https://github.com/4ertus2))
+
+### Исправления ошибок
+* Этот релиз также содержит все исправления из 19.3 и 19.1.
+* Исправлена ошибка во вторичных индексах (экспериментальная возможность): порядок гранул при INSERT был неверным. [#4407](https://github.com/yandex/ClickHouse/pull/4407) ([Nikita Vasilev](https://github.com/nikvas0))
+* Исправлена работа вторичного индекса (экспериментальная возможность) типа `set` для столбцов типа `Nullable` и `LowCardinality`. Ранее их использование вызывало ошибку `Data type must be deserialized with multiple streams` при запросе SELECT. [#4594](https://github.com/yandex/ClickHouse/pull/4594) ([Nikolai Kochetov](https://github.com/KochetovNicolai))
+* Правильное запоминание времени последнего обновления при полной перезагрузке словарей типа `executable`. [#4551](https://github.com/yandex/ClickHouse/pull/4551) ([Tema Novikov](https://github.com/temoon))
+* Исправлена неработоспособность прогресс-бара, возникшая в версии 19.3 [#4627](https://github.com/yandex/ClickHouse/pull/4627) ([filimonov](https://github.com/filimonov))
+* Исправлены неправильные значения MemoryTracker, если кусок памяти был уменьшен в размере, в очень редких случаях. [#4619](https://github.com/yandex/ClickHouse/pull/4619) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Исправлено undefined behaviour в ThreadPool [#4612](https://github.com/yandex/ClickHouse/pull/4612) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Исправлено очень редкое падение с сообщением `mutex lock failed: Invalid argument`, которое могло произойти, если таблица типа MergeTree удалялась одновременно с SELECT. [#4608](https://github.com/yandex/ClickHouse/pull/4608) ([Alex Zatelepin](https://github.com/ztlpn))
+* Совместимость ODBC драйвера с типом данных `LowCardinality` [#4381](https://github.com/yandex/ClickHouse/pull/4381) ([proller](https://github.com/proller))
+* Исправление ошибки `AIOcontextPool: Found io_event with unknown id 0` под ОС FreeBSD [#4438](https://github.com/yandex/ClickHouse/pull/4438) ([urgordeadbeef](https://github.com/urgordeadbeef))
+* Таблица `system.part_log` создавалась независимо от того, была ли она объявлена в конфигурации. [#4483](https://github.com/yandex/ClickHouse/pull/4483) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Исправлено undefined behaviour в функции `dictIsIn` для словарей типа `cache`. [#4515](https://github.com/yandex/ClickHouse/pull/4515) ([alesapin](https://github.com/alesapin))
+* Исправлен deadlock в случае, если запрос SELECT блокирует одну и ту же таблицу несколько раз (например - из разных потоков, либо при выполнении разных подзапросов) и одновременно с этим производится DDL запрос. [#4535](https://github.com/yandex/ClickHouse/pull/4535) ([Alex Zatelepin](https://github.com/ztlpn))
+* Настройка `compile_expressions` выключена по-умолчанию до тех пор, пока мы не зафиксируем исходники используемой библиотеки `LLVM` и не будем проверять её под `ASan` (сейчас библиотека LLVM берётся из системы). [#4579](https://github.com/yandex/ClickHouse/pull/4579) ([alesapin](https://github.com/alesapin))
+* Исправлено падение по `std::terminate`, если `invalidate_query` для внешних словарей с источником `clickhouse` вернул неправильный результат (пустой; более чем одну строку; более чем один столбец). Исправлена ошибка, из-за которой запрос `invalidate_query` производился каждые пять секунд, независимо от указанного `lifetime`. [#4583](https://github.com/yandex/ClickHouse/pull/4583) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Исправлен deadlock в случае, если запрос `invalidate_query` для внешнего словаря с источником `clickhouse` использовал таблицу `system.dictionaries` или базу данных типа `Dictionary` (редкий случай). [#4599](https://github.com/yandex/ClickHouse/pull/4599) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Исправлена работа CROSS JOIN с пустым WHERE [#4598](https://github.com/yandex/ClickHouse/pull/4598) ([Artem Zuikov](https://github.com/4ertus2))
+* Исправлен segfault в функции `replicate` с константным аргументом. [#4603](https://github.com/yandex/ClickHouse/pull/4603) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Исправлена работа predicate pushdown (настройка `enable_optimize_predicate_expression`) с лямбда-функциями. [#4408](https://github.com/yandex/ClickHouse/pull/4408) ([Winter Zhang](https://github.com/zhang2014))
+* Множественные исправления для множества JOIN в одном запросе. [#4595](https://github.com/yandex/ClickHouse/pull/4595) ([Artem Zuikov](https://github.com/4ertus2))
+
+### Улучшения
+* Поддержка алиасов в секции JOIN ON для правой таблицы [#4412](https://github.com/yandex/ClickHouse/pull/4412) ([Artem Zuikov](https://github.com/4ertus2))
+* Используются правильные алиасы в случае множественных JOIN с подзапросами. [#4474](https://github.com/yandex/ClickHouse/pull/4474) ([Artem Zuikov](https://github.com/4ertus2))
+* Исправлена логика работы predicate pushdown (настройка `enable_optimize_predicate_expression`) для JOIN. [#4387](https://github.com/yandex/ClickHouse/pull/4387) ([Ivan](https://github.com/abyss7))
+
+### Улучшения производительности
+* Улучшена эвристика оптимизации "перенос в PREWHERE". [#4405](https://github.com/yandex/ClickHouse/pull/4405) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Используются настоящие lookup таблицы вместо хэш-таблиц в случае 8 и 16 битных ключей. Интерфейс хэш-таблиц обобщён, чтобы поддерживать этот случай. [#4536](https://github.com/yandex/ClickHouse/pull/4536) ([Amos Bird](https://github.com/amosbird))
+* Улучшена производительность сравнения строк. [#4564](https://github.com/yandex/ClickHouse/pull/4564) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Очередь DDL операций (для запросов ON CLUSTER) очищается в отдельном потоке, чтобы не замедлять основную работу. [#4502](https://github.com/yandex/ClickHouse/pull/4502) ([Alex Zatelepin](https://github.com/ztlpn))
+* Даже если настройка `min_bytes_to_use_direct_io` выставлена в 1, не каждый файл открывался в режиме O_DIRECT, потому что размер файлов иногда недооценивался на размер одного сжатого блока. [#4526](https://github.com/yandex/ClickHouse/pull/4526) ([alexey-milovidov](https://github.com/alexey-milovidov))
+
+### Улучшения сборки/тестирования/пакетирования
+* Добавлена поддержка компилятора clang-9 [#4604](https://github.com/yandex/ClickHouse/pull/4604) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Исправлены неправильные `__asm__` инструкции [#4621](https://github.com/yandex/ClickHouse/pull/4621) ([Konstantin Podshumok](https://github.com/podshumok))
+* Добавлена поддержка задания настроек выполнения запросов для `clickhouse-performance-test` из командной строки. [#4437](https://github.com/yandex/ClickHouse/pull/4437) ([alesapin](https://github.com/alesapin))
+* Тесты словарей перенесены в интеграционные тесты. [#4477](https://github.com/yandex/ClickHouse/pull/4477) ([alesapin](https://github.com/alesapin))
+* В набор автоматизированных тестов производительности добавлены запросы, находящиеся в разделе "benchmark" на официальном сайте. [#4496](https://github.com/yandex/ClickHouse/pull/4496) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Исправления сборки в случае использования внешних библиотек lz4 и xxhash. [#4495](https://github.com/yandex/ClickHouse/pull/4495) ([Orivej Desh](https://github.com/orivej))
+* Исправлен undefined behaviour, если функция `quantileTiming` была вызвана с отрицательным или нецелым аргументом (обнаружено с помощью fuzz test под undefined behaviour sanitizer). [#4506](https://github.com/yandex/ClickHouse/pull/4506) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Исправлены опечатки в коде. [#4531](https://github.com/yandex/ClickHouse/pull/4531) ([sdk2](https://github.com/sdk2))
+* Исправлена сборка под Mac. [#4371](https://github.com/yandex/ClickHouse/pull/4371) ([Vitaly Baranov](https://github.com/vitlibar))
+* Исправлена сборка под FreeBSD и для некоторых необычных конфигурациях сборки. [#4444](https://github.com/yandex/ClickHouse/pull/4444) ([proller](https://github.com/proller))
+
+
+## ClickHouse release 19.3.7, 2019-03-12
+
+### Исправления ошибок
+
+* Исправлена ошибка в #3920. Ошибка проявлялась в виде случайных повреждений кэша (сообщения `Unknown codec family code`, `Cannot seek through file`) и segfault. Ошибка впервые возникла в 19.1 и присутствует во всех версиях до 19.1.10 и 19.3.6. [#4623](https://github.com/yandex/ClickHouse/pull/4623) ([alexey-milovidov](https://github.com/alexey-milovidov))
+
+
+## ClickHouse release 19.3.6, 2019-03-02
+
+### Исправления ошибок
+
+* Если в пуле потоков было более 1000 потоков, то при выходе из потока, вызывается `std::terminate`. [Azat Khuzhin](https://github.com/azat) [#4485](https://github.com/yandex/ClickHouse/pull/4485) [#4505](https://github.com/yandex/ClickHouse/pull/4505) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Теперь возможно создавать таблицы `ReplicatedMergeTree*` с комментариями столбцов без указания DEFAULT, а также с CODEC но без COMMENT и DEFAULT. Исправлено сравнение CODEC друг с другом. [#4523](https://github.com/yandex/ClickHouse/pull/4523) ([alesapin](https://github.com/alesapin))
+* Исправлено падение при JOIN по массивам и кортежам. [#4552](https://github.com/yandex/ClickHouse/pull/4552) ([Artem Zuikov](https://github.com/4ertus2))
+* Исправлено падение `clickhouse-copier` с сообщением `ThreadStatus not created`. [#4540](https://github.com/yandex/ClickHouse/pull/4540) ([Artem Zuikov](https://github.com/4ertus2))
+* Исправлено зависание сервера при завершении работы в случае использования распределённых DDL. [#4472](https://github.com/yandex/ClickHouse/pull/4472) ([Alex Zatelepin](https://github.com/ztlpn))
+* В сообщениях об ошибке при парсинге текстовых форматов, выдавались неправильные номера столбцов, в случае, если номер больше 10. [#4484](https://github.com/yandex/ClickHouse/pull/4484) ([alexey-milovidov](https://github.com/alexey-milovidov))
+
+### Улучшения сборки/тестирования/пакетирования
+
+* Исправлена сборка с включенным AVX. [#4527](https://github.com/yandex/ClickHouse/pull/4527) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Исправлена поддержка расширенных метрик выполнения запроса в случае, если ClickHouse был собран на системе с новым ядром Linux, а запускается на системе с существенно более старым ядром. [#4541](https://github.com/yandex/ClickHouse/pull/4541) ([nvartolomei](https://github.com/nvartolomei))
+* Продолжение работы в случае невозможности применить настройку `core_dump.size_limit` с выводом предупреждения. [#4473](https://github.com/yandex/ClickHouse/pull/4473) ([proller](https://github.com/proller))
+* Удалено `inline` для `void readBinary(...)` в `Field.cpp`. [#4530](https://github.com/yandex/ClickHouse/pull/4530) ([hcz](https://github.com/hczhcz))
+
+
## ClickHouse release 19.3.5, 2019-02-21
### Исправления ошибок:
@@ -74,7 +160,7 @@
* Исправлена ошибка, из-за которой при запросе к таблице `system.tables` могло возникать исключение `table doesn't exist`. [#4313](https://github.com/yandex/ClickHouse/pull/4313) ([alexey-milovidov](https://github.com/alexey-milovidov))
* Исправлена ошибка, приводившая к падению `clickhouse-client` в интерактивном режиме, если успеть выйти из него во время загрузки подсказок командной строки. [#4317](https://github.com/yandex/ClickHouse/pull/4317) ([alexey-milovidov](https://github.com/alexey-milovidov))
* Исправлена ошибка, приводившая к неверным результатам исполнения мутаций, содержащих оператор `IN`. [#4099](https://github.com/yandex/ClickHouse/pull/4099) ([Alex Zatelepin](https://github.com/ztlpn))
-* Исправлена ошибка, из-за которой, если была создана база данных с движком `Dictionary`, все словари загружались при старте сервера, а словари с источником из локального ClickHouse не могли загрузиться. [#4255](https://github.com/yandex/ClickHouse/pull/4255) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Исправлена ошибка, из-за которой, если была создана база данных с движком `Dictionary`, все словари загружались при старте сервера, а словари с источником из локального ClickHouse не могли загрузиться. [#4255](https://github.com/yandex/ClickHouse/pull/4255) ([alexey-milovidov](https://github.com/alexey-milovidov))
* Исправлено повторное создание таблиц с системными логами (`system.query_log`, `system.part_log`) при остановке сервера. [#4254](https://github.com/yandex/ClickHouse/pull/4254) ([alexey-milovidov](https://github.com/alexey-milovidov))
* Исправлен вывод типа возвращаемого значения, а также использование блокировок в функции `joinGet`. [#4153](https://github.com/yandex/ClickHouse/pull/4153) ([Amos Bird](https://github.com/amosbird))
* Исправлено падение сервера при использовании настройки `allow_experimental_multiple_joins_emulation`. [52de2c](https://github.com/yandex/ClickHouse/commit/52de2cd927f7b5257dd67e175f0a5560a48840d0) ([Artem Zuikov](https://github.com/4ertus2))
@@ -98,7 +184,7 @@
* Добавлен инструмент, собирающий changelog из описаний pull request-ов. [#4169](https://github.com/yandex/ClickHouse/pull/4169) [#4173](https://github.com/yandex/ClickHouse/pull/4173) ([KochetovNicolai](https://github.com/KochetovNicolai)) ([KochetovNicolai](https://github.com/KochetovNicolai))
* Добавлен puppet-модуль для Clickhouse. [#4182](https://github.com/yandex/ClickHouse/pull/4182) ([Maxim Fedotov](https://github.com/MaxFedotov))
* Добавлена документация для нескольких недокументированных функций. [#4168](https://github.com/yandex/ClickHouse/pull/4168) ([Winter Zhang](https://github.com/zhang2014))
-* Исправления сборки под ARM. [#4210](https://github.com/yandex/ClickHouse/pull/4210)[#4306](https://github.com/yandex/ClickHouse/pull/4306) [#4291](https://github.com/yandex/ClickHouse/pull/4291) ([proller](https://github.com/proller)) ([proller](https://github.com/proller))
+* Исправления сборки под ARM. [#4210](https://github.com/yandex/ClickHouse/pull/4210)[#4306](https://github.com/yandex/ClickHouse/pull/4306) [#4291](https://github.com/yandex/ClickHouse/pull/4291) ([proller](https://github.com/proller)) ([proller](https://github.com/proller))
* Добавлена возможность запускать тесты словарей из `ctest`. [#4189](https://github.com/yandex/ClickHouse/pull/4189) ([proller](https://github.com/proller))
* Теперь директорией с SSL-сертификатами по умолчанию является `/etc/ssl`. [#4167](https://github.com/yandex/ClickHouse/pull/4167) ([alexey-milovidov](https://github.com/alexey-milovidov))
* Добавлена проверка доступности SSE и AVX-инструкций на старте. [#4234](https://github.com/yandex/ClickHouse/pull/4234) ([Igr](https://github.com/igron99))
@@ -133,6 +219,18 @@
* Уменьшено время ожидания завершения сервера и завершения запросов `ALTER`. [#4372](https://github.com/yandex/ClickHouse/pull/4372) ([alexey-milovidov](https://github.com/alexey-milovidov))
* Добавлена информация о значении настройки `replicated_can_become_leader` в таблицу `system.replicas`. Добавлено логирование в случае, если реплика не собирается стать лидером. [#4379](https://github.com/yandex/ClickHouse/pull/4379) ([Alex Zatelepin](https://github.com/ztlpn))
+## ClickHouse release 19.1.14, 2019-03-14
+
+* Исправлена ошибка `Column ... queried more than once`, которая могла произойти в случае включенной настройки `asterisk_left_columns_only` в случае использования `GLOBAL JOIN` а также `SELECT *` (редкий случай). Эта ошибка изначально отсутствует в версиях 19.3 и более новых. [6bac7d8d](https://github.com/yandex/ClickHouse/pull/4692/commits/6bac7d8d11a9b0d6de0b32b53c47eb2f6f8e7062) ([Artem Zuikov](https://github.com/4ertus2))
+
+## ClickHouse release 19.1.13, 2019-03-12
+
+Этот релиз содержит такие же исправления ошибок, как и 19.3.7.
+
+## ClickHouse release 19.1.10, 2019-03-03
+
+Этот релиз содержит такие же исправления ошибок, как и 19.3.6.
+
## ClickHouse release 19.1.9, 2019-02-21
### Исправления ошибок:
@@ -152,7 +250,7 @@
* Исправлен вывод типа возвращаемого значения, а также использование блокировок в функции `joinGet`. [#4153](https://github.com/yandex/ClickHouse/pull/4153) ([Amos Bird](https://github.com/amosbird))
* Исправлено повторное создание таблиц с системными логами (`system.query_log`, `system.part_log`) при остановке сервера. [#4254](https://github.com/yandex/ClickHouse/pull/4254) ([alexey-milovidov](https://github.com/alexey-milovidov))
-* Исправлена ошибка, из-за которой, если была создана база данных с движком `Dictionary`, все словари загружались при старте сервера, а словари с источником из локального ClickHouse не могли загрузиться. [#4255](https://github.com/yandex/ClickHouse/pull/4255) ([alexey-milovidov](https://github.com/alexey-milovidov))
+* Исправлена ошибка, из-за которой, если была создана база данных с движком `Dictionary`, все словари загружались при старте сервера, а словари с источником из локального ClickHouse не могли загрузиться. [#4255](https://github.com/yandex/ClickHouse/pull/4255) ([alexey-milovidov](https://github.com/alexey-milovidov))
* Исправлена ошибка, приводившая к неверным результатам исполнения мутаций, содержащих оператор `IN`. [#4099](https://github.com/yandex/ClickHouse/pull/4099) ([Alex Zatelepin](https://github.com/ztlpn))
* Исправлена ошибка, приводившая к падению `clickhouse-client` в интерактивном режиме, если успеть выйти из него во время загрузки подсказок командной строки. [#4317](https://github.com/yandex/ClickHouse/pull/4317) ([alexey-milovidov](https://github.com/alexey-milovidov))
* Исправлена ошибка, из-за которой при запросе к таблице `system.tables` могло возникать исключение `table doesn't exist`. [#4313](https://github.com/yandex/ClickHouse/pull/4313) ([alexey-milovidov](https://github.com/alexey-milovidov))
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 8c0627ce569..45ee1dfbb41 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -1,7 +1,11 @@
-project (ClickHouse)
-cmake_minimum_required (VERSION 3.3)
-
+project(ClickHouse)
+cmake_minimum_required(VERSION 3.3)
+cmake_policy(SET CMP0023 NEW)
set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} "${CMAKE_CURRENT_SOURCE_DIR}/cmake/Modules/")
+set(CMAKE_EXPORT_COMPILE_COMMANDS 1) # Write compile_commands.json
+set(CMAKE_LINK_DEPENDS_NO_SHARED 1) # Do not relink all depended targets on .so
+set(CMAKE_CONFIGURATION_TYPES "RelWithDebInfo;Debug;Release;MinSizeRel" CACHE STRING "" FORCE)
+set(CMAKE_DEBUG_POSTFIX "d" CACHE STRING "Generate debug library name with a postfix.") # To be consistent with CMakeLists from contrib libs.
option(ENABLE_IPO "Enable inter-procedural optimization (aka LTO)" OFF) # need cmake 3.9+
if(ENABLE_IPO)
@@ -37,9 +41,6 @@ if (EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/.git" AND NOT EXISTS "${ClickHouse_SOURC
message (FATAL_ERROR "Submodules are not initialized. Run\n\tgit submodule update --init --recursive")
endif ()
-# Write compile_commands.json
-set(CMAKE_EXPORT_COMPILE_COMMANDS 1)
-
include (cmake/find_ccache.cmake)
if (NOT CMAKE_BUILD_TYPE OR CMAKE_BUILD_TYPE STREQUAL "None")
@@ -49,8 +50,6 @@ endif ()
string(TOUPPER ${CMAKE_BUILD_TYPE} CMAKE_BUILD_TYPE_UC)
message (STATUS "CMAKE_BUILD_TYPE: ${CMAKE_BUILD_TYPE}")
-set (CMAKE_CONFIGURATION_TYPES "RelWithDebInfo;Debug;Release;MinSizeRel" CACHE STRING "" FORCE)
-set (CMAKE_DEBUG_POSTFIX "d" CACHE STRING "Generate debug library name with a postfix.") # To be consistent with CMakeLists from contrib libs.
option (USE_STATIC_LIBRARIES "Set to FALSE to use shared libraries" ON)
option (MAKE_STATIC_LIBRARIES "Set to FALSE to make shared libraries" ${USE_STATIC_LIBRARIES})
@@ -141,7 +140,7 @@ if(NOT COMPILER_CLANG) # clang: error: the clang compiler does not support '-mar
endif()
if (ARCH_NATIVE)
- set (COMPILER_FLAGS "${COMPILER_FLAGS} -march=native")
+ set (COMPILER_FLAGS "${COMPILER_FLAGS} -march=native")
endif ()
# Special options for better optimized code with clang
@@ -179,14 +178,20 @@ include (cmake/use_libcxx.cmake)
# This is intended for more control of what we are linking.
set (DEFAULT_LIBS "")
-if (OS_LINUX AND NOT UNBUNDLED)
- # Note: this probably has no effict, but I'm not an expert in CMake.
+if (OS_LINUX AND NOT UNBUNDLED AND (GLIBC_COMPATIBILITY OR USE_LIBCXX))
+ # Note: this probably has no effect, but I'm not an expert in CMake.
set (CMAKE_C_IMPLICIT_LINK_LIBRARIES "")
set (CMAKE_CXX_IMPLICIT_LINK_LIBRARIES "")
# Disable default linked libraries.
set (DEFAULT_LIBS "-nodefaultlibs")
+ # We need builtins from Clang's RT even without libcxx - for ubsan+int128. See https://bugs.llvm.org/show_bug.cgi?id=16404
+ set (BUILTINS_LIB_PATH "")
+ if (COMPILER_CLANG)
+ execute_process (COMMAND ${CMAKE_CXX_COMPILER} --print-file-name=libclang_rt.builtins-${CMAKE_SYSTEM_PROCESSOR}.a OUTPUT_VARIABLE BUILTINS_LIB_PATH OUTPUT_STRIP_TRAILING_WHITESPACE)
+ endif ()
+
# Add C++ libraries.
#
# This consist of:
@@ -197,14 +202,9 @@ if (OS_LINUX AND NOT UNBUNDLED)
#
# There are two variants of C++ library: libc++ (from LLVM compiler infrastructure) and libstdc++ (from GCC).
if (USE_LIBCXX)
- set (BUILTINS_LIB_PATH "")
- if (COMPILER_CLANG)
- execute_process (COMMAND ${CMAKE_CXX_COMPILER} --print-file-name=libclang_rt.builtins-${CMAKE_SYSTEM_PROCESSOR}.a OUTPUT_VARIABLE BUILTINS_LIB_PATH OUTPUT_STRIP_TRAILING_WHITESPACE)
- endif ()
-
set (DEFAULT_LIBS "${DEFAULT_LIBS} -Wl,-Bstatic -lc++ -lc++abi -lgcc_eh ${BUILTINS_LIB_PATH} -Wl,-Bdynamic")
else ()
- set (DEFAULT_LIBS "${DEFAULT_LIBS} -Wl,-Bstatic -lstdc++ -lgcc_eh -lgcc -Wl,-Bdynamic")
+ set (DEFAULT_LIBS "${DEFAULT_LIBS} -Wl,-Bstatic -lstdc++ -lgcc_eh -lgcc ${BUILTINS_LIB_PATH} -Wl,-Bdynamic")
endif ()
# Linking with GLIBC prevents portability of binaries to older systems.
@@ -216,6 +216,7 @@ if (OS_LINUX AND NOT UNBUNDLED)
string (TOUPPER ${CMAKE_BUILD_TYPE} CMAKE_BUILD_TYPE_UC)
set (CMAKE_POSTFIX_VARIABLE "CMAKE_${CMAKE_BUILD_TYPE_UC}_POSTFIX")
+ # FIXME: glibc-compatibility may be non-static in some builds!
set (DEFAULT_LIBS "${DEFAULT_LIBS} libs/libglibc-compatibility/libglibc-compatibility${${CMAKE_POSTFIX_VARIABLE}}.a")
endif ()
@@ -227,6 +228,11 @@ if (OS_LINUX AND NOT UNBUNDLED)
message(STATUS "Default libraries: ${DEFAULT_LIBS}")
endif ()
+if (DEFAULT_LIBS)
+ # Add default libs to all targets as the last dependency.
+ set(CMAKE_CXX_STANDARD_LIBRARIES ${DEFAULT_LIBS})
+endif ()
+
if (NOT MAKE_STATIC_LIBRARIES)
set(CMAKE_POSITION_INDEPENDENT_CODE ON)
@@ -310,6 +316,8 @@ include (cmake/find_pdqsort.cmake)
include (cmake/find_hdfs3.cmake) # uses protobuf
include (cmake/find_consistent-hashing.cmake)
include (cmake/find_base64.cmake)
+include (cmake/find_hyperscan.cmake)
+include (cmake/find_lfalloc.cmake)
find_contrib_lib(cityhash)
find_contrib_lib(farmhash)
find_contrib_lib(metrohash)
@@ -336,35 +344,29 @@ add_subdirectory (dbms)
include (cmake/print_include_directories.cmake)
-
-if (DEFAULT_LIBS)
- # Add default libs to all targets as the last dependency.
- # I have found no better way to specify default libs in CMake that will appear single time in specific order at the end of linker arguments.
-
- function(add_default_libs target_name)
+if (GLIBC_COMPATIBILITY)
+ # FIXME: actually glibc-compatibility should always be built first,
+ # because it's unconditionally linked via $DEFAULT_LIBS,
+ # and these looks like the first places that get linked.
+ function (add_glibc_compat target_name)
if (TARGET ${target_name})
- # message(STATUS "Has target ${target_name}")
- set_property(TARGET ${target_name} APPEND PROPERTY LINK_LIBRARIES "${DEFAULT_LIBS}")
- set_property(TARGET ${target_name} APPEND PROPERTY INTERFACE_LINK_LIBRARIES "${DEFAULT_LIBS}")
- if (GLIBC_COMPATIBILITY)
- add_dependencies(${target_name} glibc-compatibility)
- endif ()
+ add_dependencies(${target_name} glibc-compatibility)
endif ()
endfunction ()
- add_default_libs(ltdl)
- add_default_libs(zlibstatic)
- add_default_libs(jemalloc)
- add_default_libs(unwind)
- add_default_libs(memcpy)
- add_default_libs(Foundation)
- add_default_libs(common)
- add_default_libs(gtest)
- add_default_libs(lz4)
- add_default_libs(zstd)
- add_default_libs(snappy)
- add_default_libs(arrow)
- add_default_libs(protoc)
- add_default_libs(thrift_static)
- add_default_libs(boost_regex_internal)
+ add_glibc_compat(ltdl)
+ add_glibc_compat(zlibstatic)
+ add_glibc_compat(jemalloc)
+ add_glibc_compat(unwind)
+ add_glibc_compat(memcpy)
+ add_glibc_compat(Foundation)
+ add_glibc_compat(common)
+ add_glibc_compat(gtest)
+ add_glibc_compat(lz4)
+ add_glibc_compat(zstd)
+ add_glibc_compat(snappy)
+ add_glibc_compat(arrow)
+ add_glibc_compat(protoc)
+ add_glibc_compat(thrift_static)
+ add_glibc_compat(boost_regex_internal)
endif ()
diff --git a/README.md b/README.md
index 3e840d2cf10..02a50be007b 100644
--- a/README.md
+++ b/README.md
@@ -12,5 +12,8 @@ ClickHouse is an open-source column-oriented database management system that all
* You can also [fill this form](https://forms.yandex.com/surveys/meet-yandex-clickhouse-team/) to meet Yandex ClickHouse team in person.
## Upcoming Events
-
-* [ClickHouse Community Meetup](https://www.eventbrite.com/e/clickhouse-meetup-in-madrid-registration-55376746339) in Madrid on April 2.
+* [ClickHouse Community Meetup in Limassol](https://www.facebook.com/events/386638262181785/) on May 7.
+* ClickHouse at [Percona Live 2019](https://www.percona.com/live/19/other-open-source-databases-track) in Austin on May 28-30.
+* [ClickHouse Community Meetup in Beijing](https://www.huodongxing.com/event/2483759276200) on June 8.
+* [ClickHouse Community Meetup in Shenzhen](https://www.huodongxing.com/event/3483759917300) on October 20.
+* [ClickHouse Community Meetup in Shanghai](https://www.huodongxing.com/event/4483760336000) on October 27.
diff --git a/ci/jobs/quick-build/run.sh b/ci/jobs/quick-build/run.sh
index 6a948c560ee..9e8fe9353d6 100755
--- a/ci/jobs/quick-build/run.sh
+++ b/ci/jobs/quick-build/run.sh
@@ -21,7 +21,7 @@ BUILD_TARGETS=clickhouse
BUILD_TYPE=Debug
ENABLE_EMBEDDED_COMPILER=0
-CMAKE_FLAGS="-D CMAKE_C_FLAGS_ADD=-g0 -D CMAKE_CXX_FLAGS_ADD=-g0 -D ENABLE_JEMALLOC=0 -D ENABLE_CAPNP=0 -D ENABLE_RDKAFKA=0 -D ENABLE_UNWIND=0 -D ENABLE_ICU=0 -D ENABLE_POCO_MONGODB=0 -D ENABLE_POCO_NETSSL=0 -D ENABLE_POCO_ODBC=0 -D ENABLE_ODBC=0 -D ENABLE_MYSQL=0"
+CMAKE_FLAGS="-D CMAKE_C_FLAGS_ADD=-g0 -D CMAKE_CXX_FLAGS_ADD=-g0 -D ENABLE_JEMALLOC=0 -D ENABLE_CAPNP=0 -D ENABLE_RDKAFKA=0 -D ENABLE_UNWIND=0 -D ENABLE_ICU=0 -D ENABLE_POCO_MONGODB=0 -D ENABLE_POCO_NETSSL=0 -D ENABLE_POCO_ODBC=0 -D ENABLE_ODBC=0 -D ENABLE_MYSQL=0 -D ENABLE_SSL=0 -D ENABLE_POCO_NETSSL=0"
[[ $(uname) == "FreeBSD" ]] && COMPILER_PACKAGE_VERSION=devel && export COMPILER_PATH=/usr/local/bin
diff --git a/cmake/find_execinfo.cmake b/cmake/find_execinfo.cmake
index 650d279983c..85cc5cf951a 100644
--- a/cmake/find_execinfo.cmake
+++ b/cmake/find_execinfo.cmake
@@ -1,9 +1,8 @@
if (OS_FREEBSD)
find_library (EXECINFO_LIBRARY execinfo)
find_library (ELF_LIBRARY elf)
- message (STATUS "Using execinfo: ${EXECINFO_LIBRARY}")
- message (STATUS "Using elf: ${ELF_LIBRARY}")
+ set (EXECINFO_LIBRARIES ${EXECINFO_LIBRARY} ${ELF_LIBRARY})
+ message (STATUS "Using execinfo: ${EXECINFO_LIBRARIES}")
else ()
- set (EXECINFO_LIBRARY "")
- set (ELF_LIBRARY "")
+ set (EXECINFO_LIBRARIES "")
endif ()
diff --git a/cmake/find_hyperscan.cmake b/cmake/find_hyperscan.cmake
new file mode 100644
index 00000000000..a3e0b6bc9bc
--- /dev/null
+++ b/cmake/find_hyperscan.cmake
@@ -0,0 +1,33 @@
+if (HAVE_SSSE3)
+ option (ENABLE_HYPERSCAN "Enable hyperscan" ON)
+endif ()
+
+if (ENABLE_HYPERSCAN)
+
+option (USE_INTERNAL_HYPERSCAN_LIBRARY "Set to FALSE to use system hyperscan instead of the bundled" ${NOT_UNBUNDLED})
+
+if (NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/hyperscan/CMakeLists.txt")
+ if (USE_INTERNAL_HYPERSCAN_LIBRARY)
+ message (WARNING "submodule contrib/hyperscan is missing. to fix try run: \n git submodule update --init --recursive")
+ endif ()
+ set (MISSING_INTERNAL_HYPERSCAN_LIBRARY 1)
+ set (USE_INTERNAL_HYPERSCAN_LIBRARY 0)
+endif ()
+
+if (NOT USE_INTERNAL_HYPERSCAN_LIBRARY)
+ find_library (HYPERSCAN_LIBRARY hs)
+ find_path (HYPERSCAN_INCLUDE_DIR NAMES hs/hs.h hs.h PATHS ${HYPERSCAN_INCLUDE_PATHS})
+endif ()
+
+if (HYPERSCAN_LIBRARY AND HYPERSCAN_INCLUDE_DIR)
+ set (USE_HYPERSCAN 1)
+elseif (NOT MISSING_INTERNAL_HYPERSCAN_LIBRARY)
+ set (HYPERSCAN_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/contrib/hyperscan/src)
+ set (HYPERSCAN_LIBRARY hs)
+ set (USE_HYPERSCAN 1)
+ set (USE_INTERNAL_HYPERSCAN_LIBRARY 1)
+endif()
+
+message (STATUS "Using hyperscan=${USE_HYPERSCAN}: ${HYPERSCAN_INCLUDE_DIR} : ${HYPERSCAN_LIBRARY}")
+
+endif ()
diff --git a/cmake/find_lfalloc.cmake b/cmake/find_lfalloc.cmake
new file mode 100644
index 00000000000..c9b2ce5d436
--- /dev/null
+++ b/cmake/find_lfalloc.cmake
@@ -0,0 +1,10 @@
+if (NOT SANITIZE AND NOT ARCH_ARM AND NOT ARCH_32 AND NOT ARCH_PPC64LE AND NOT OS_FREEBSD)
+ option (ENABLE_LFALLOC "Set to FALSE to use system libgsasl library instead of bundled" ${NOT_UNBUNDLED})
+endif ()
+
+if (ENABLE_LFALLOC)
+ set (USE_LFALLOC 1)
+ set (USE_LFALLOC_RANDOM_HINT 1)
+ set (LFALLOC_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/contrib/lfalloc/src)
+ message (STATUS "Using lfalloc=${USE_LFALLOC}: ${LFALLOC_INCLUDE_DIR}")
+endif ()
diff --git a/cmake/find_poco.cmake b/cmake/find_poco.cmake
index 012f269d48d..07d81c1bbe8 100644
--- a/cmake/find_poco.cmake
+++ b/cmake/find_poco.cmake
@@ -36,6 +36,8 @@ elseif (NOT MISSING_INTERNAL_POCO_LIBRARY)
set (ENABLE_DATA_SQLITE 0 CACHE BOOL "")
set (ENABLE_DATA_MYSQL 0 CACHE BOOL "")
set (ENABLE_DATA_POSTGRESQL 0 CACHE BOOL "")
+ set (ENABLE_ENCODINGS 0 CACHE BOOL "")
+
# new after 2.0.0:
set (POCO_ENABLE_ZIP 0 CACHE BOOL "")
set (POCO_ENABLE_PAGECOMPILER 0 CACHE BOOL "")
diff --git a/cmake/find_rdkafka.cmake b/cmake/find_rdkafka.cmake
index ff3360baa74..3363c657f91 100644
--- a/cmake/find_rdkafka.cmake
+++ b/cmake/find_rdkafka.cmake
@@ -1,5 +1,5 @@
# Freebsd: contrib/cppkafka/include/cppkafka/detail/endianness.h:53:23: error: 'betoh16' was not declared in this scope
-if (NOT ARCH_ARM AND NOT ARCH_32 AND NOT APPLE AND NOT OS_FREEBSD)
+if (NOT ARCH_ARM AND NOT ARCH_32 AND NOT APPLE AND NOT OS_FREEBSD AND OPENSSL_FOUND)
option (ENABLE_RDKAFKA "Enable kafka" ON)
endif ()
diff --git a/cmake/find_ssl.cmake b/cmake/find_ssl.cmake
index 4af11e033fe..6d65ccc66bf 100644
--- a/cmake/find_ssl.cmake
+++ b/cmake/find_ssl.cmake
@@ -1,7 +1,19 @@
+option (ENABLE_SSL "Enable ssl" ON)
+
+if (ENABLE_SSL)
+
if(NOT ARCH_32)
option(USE_INTERNAL_SSL_LIBRARY "Set to FALSE to use system *ssl library instead of bundled" ${NOT_UNBUNDLED})
endif()
+if(NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/ssl/CMakeLists.txt")
+ if(USE_INTERNAL_SSL_LIBRARY)
+ message(WARNING "submodule contrib/ssl is missing. to fix try run: \n git submodule update --init --recursive")
+ endif()
+ set(USE_INTERNAL_SSL_LIBRARY 0)
+ set(MISSING_INTERNAL_SSL_LIBRARY 1)
+endif()
+
set (OPENSSL_USE_STATIC_LIBS ${USE_STATIC_LIBRARIES})
if (NOT USE_INTERNAL_SSL_LIBRARY)
@@ -28,7 +40,7 @@ if (NOT USE_INTERNAL_SSL_LIBRARY)
endif ()
endif ()
-if (NOT OPENSSL_FOUND)
+if (NOT OPENSSL_FOUND AND NOT MISSING_INTERNAL_SSL_LIBRARY)
set (USE_INTERNAL_SSL_LIBRARY 1)
set (OPENSSL_ROOT_DIR "${ClickHouse_SOURCE_DIR}/contrib/ssl")
set (OPENSSL_INCLUDE_DIR "${OPENSSL_ROOT_DIR}/include")
@@ -43,4 +55,11 @@ if (NOT OPENSSL_FOUND)
set (OPENSSL_FOUND 1)
endif ()
-message (STATUS "Using ssl=${OPENSSL_FOUND}: ${OPENSSL_INCLUDE_DIR} : ${OPENSSL_LIBRARIES}")
+if(OPENSSL_FOUND)
+ # we need keep OPENSSL_FOUND for many libs in contrib
+ set(USE_SSL 1)
+endif()
+
+endif ()
+
+message (STATUS "Using ssl=${USE_SSL}: ${OPENSSL_INCLUDE_DIR} : ${OPENSSL_LIBRARIES}")
diff --git a/contrib/CMakeLists.txt b/contrib/CMakeLists.txt
index fe95dcad041..447ff6fb880 100644
--- a/contrib/CMakeLists.txt
+++ b/contrib/CMakeLists.txt
@@ -4,7 +4,7 @@ if (CMAKE_CXX_COMPILER_ID STREQUAL "GNU")
set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wno-unused-function -Wno-unused-variable -Wno-unused-but-set-variable -Wno-unused-result -Wno-deprecated-declarations -Wno-maybe-uninitialized -Wno-format -Wno-misleading-indentation -Wno-stringop-overflow -Wno-implicit-function-declaration -Wno-return-type -Wno-array-bounds -Wno-bool-compare -Wno-int-conversion -Wno-switch")
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-old-style-cast -Wno-unused-function -Wno-unused-variable -Wno-unused-but-set-variable -Wno-unused-result -Wno-deprecated-declarations -Wno-non-virtual-dtor -Wno-maybe-uninitialized -Wno-format -Wno-misleading-indentation -Wno-implicit-fallthrough -Wno-class-memaccess -Wno-sign-compare -std=c++1z")
elseif (CMAKE_CXX_COMPILER_ID STREQUAL "Clang")
- set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wno-unused-function -Wno-unused-variable -Wno-unused-result -Wno-deprecated-declarations -Wno-format -Wno-parentheses-equality -Wno-tautological-constant-compare -Wno-tautological-constant-out-of-range-compare -Wno-implicit-function-declaration -Wno-return-type -Wno-pointer-bool-conversion -Wno-enum-conversion -Wno-int-conversion -Wno-switch")
+ set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wno-unused-function -Wno-unused-variable -Wno-unused-result -Wno-deprecated-declarations -Wno-format -Wno-parentheses-equality -Wno-tautological-constant-compare -Wno-tautological-constant-out-of-range-compare -Wno-implicit-function-declaration -Wno-return-type -Wno-pointer-bool-conversion -Wno-enum-conversion -Wno-int-conversion -Wno-switch -Wno-string-plus-int")
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-old-style-cast -Wno-unused-function -Wno-unused-variable -Wno-unused-result -Wno-deprecated-declarations -Wno-non-virtual-dtor -Wno-format -Wno-inconsistent-missing-override -std=c++1z")
endif ()
@@ -125,13 +125,17 @@ endif ()
if (ENABLE_MYSQL AND USE_INTERNAL_MYSQL_LIBRARY)
add_subdirectory (mariadb-connector-c-cmake)
target_include_directories(mysqlclient BEFORE PRIVATE ${ZLIB_INCLUDE_DIR})
- target_include_directories(mysqlclient BEFORE PRIVATE ${OPENSSL_INCLUDE_DIR})
+ if(OPENSSL_INCLUDE_DIR)
+ target_include_directories(mysqlclient BEFORE PRIVATE ${OPENSSL_INCLUDE_DIR})
+ endif()
endif ()
if (USE_INTERNAL_RDKAFKA_LIBRARY)
add_subdirectory (librdkafka-cmake)
target_include_directories(rdkafka BEFORE PRIVATE ${ZLIB_INCLUDE_DIR})
- target_include_directories(rdkafka BEFORE PRIVATE ${OPENSSL_INCLUDE_DIR})
+ if(OPENSSL_INCLUDE_DIR)
+ target_include_directories(rdkafka BEFORE PRIVATE ${OPENSSL_INCLUDE_DIR})
+ endif()
endif ()
if (USE_RDKAFKA)
@@ -280,12 +284,17 @@ endif ()
if (USE_INTERNAL_BROTLI_LIBRARY)
add_subdirectory(brotli-cmake)
+ target_compile_definitions(brotli PRIVATE BROTLI_BUILD_PORTABLE=1)
endif ()
if (USE_INTERNAL_PROTOBUF_LIBRARY)
- set(protobuf_BUILD_TESTS OFF CACHE INTERNAL "" FORCE)
- set(protobuf_BUILD_SHARED_LIBS OFF CACHE INTERNAL "" FORCE)
+ if (MAKE_STATIC_LIBRARIES)
+ set(protobuf_BUILD_SHARED_LIBS OFF CACHE INTERNAL "" FORCE)
+ else ()
+ set(protobuf_BUILD_SHARED_LIBS ON CACHE INTERNAL "" FORCE)
+ endif ()
set(protobuf_WITH_ZLIB 0 CACHE INTERNAL "" FORCE) # actually will use zlib, but skip find
+ set(protobuf_BUILD_TESTS OFF CACHE INTERNAL "" FORCE)
add_subdirectory(protobuf/cmake)
endif ()
@@ -296,3 +305,7 @@ endif ()
if (USE_BASE64)
add_subdirectory (base64-cmake)
endif()
+
+if (USE_INTERNAL_HYPERSCAN_LIBRARY)
+ add_subdirectory (hyperscan)
+endif()
diff --git a/contrib/boost b/contrib/boost
index 6a96e8b59f7..471ea208abb 160000
--- a/contrib/boost
+++ b/contrib/boost
@@ -1 +1 @@
-Subproject commit 6a96e8b59f76148eb8ad54a9d15259f8ce84c606
+Subproject commit 471ea208abb92a5cba7d3a08a819bb728f27e95f
diff --git a/contrib/hyperscan b/contrib/hyperscan
new file mode 160000
index 00000000000..05b0f9064cc
--- /dev/null
+++ b/contrib/hyperscan
@@ -0,0 +1 @@
+Subproject commit 05b0f9064cca4bd55548dedb0a32ed9461146c1e
diff --git a/contrib/lfalloc/src/lf_allocX64.h b/contrib/lfalloc/src/lf_allocX64.h
new file mode 100644
index 00000000000..2c4cf3f1021
--- /dev/null
+++ b/contrib/lfalloc/src/lf_allocX64.h
@@ -0,0 +1,1813 @@
+#pragma once
+
+#include
+#include
+#include
+
+#include "lfmalloc.h"
+
+#include "util/system/compiler.h"
+#include "util/system/types.h"
+#include
+
+#ifdef _MSC_VER
+#ifndef _CRT_SECURE_NO_WARNINGS
+#define _CRT_SECURE_NO_WARNINGS
+#endif
+#ifdef _M_X64
+#define _64_
+#endif
+#include
+#define WIN32_LEAN_AND_MEAN
+#include
+#pragma intrinsic(_InterlockedCompareExchange)
+#pragma intrinsic(_InterlockedExchangeAdd)
+
+#include
+#include
+#include
+
+#define PERTHREAD __declspec(thread)
+#define _win_
+#define Y_FORCE_INLINE __forceinline
+
+using TAtomic = volatile long;
+
+static inline long AtomicAdd(TAtomic& a, long b) {
+ return _InterlockedExchangeAdd(&a, b) + b;
+}
+
+static inline long AtomicSub(TAtomic& a, long b) {
+ return AtomicAdd(a, -b);
+}
+
+#define Y_ASSERT_NOBT(x) ((void)0)
+
+#else
+
+#include "util/system/defaults.h"
+#include "util/system/atomic.h"
+#include
+
+#if !defined(NDEBUG) && !defined(__GCCXML__)
+#define Y_ASSERT_NOBT(a) \
+ do { \
+ if (Y_UNLIKELY(!(a))) { \
+ assert(false && (a)); \
+ } \
+ } while (0)
+#else
+#define Y_ASSERT_NOBT(a) \
+ do { \
+ if (false) { \
+ bool __xxx = static_cast(a); \
+ Y_UNUSED(__xxx); \
+ } \
+ } while (0)
+#endif
+
+#include
+#include
+#include
+#include
+#include
+#include
+
+#if defined(_linux_)
+#if !defined(MADV_HUGEPAGE)
+#define MADV_HUGEPAGE 14
+#endif
+#if !defined(MAP_HUGETLB)
+#define MAP_HUGETLB 0x40000
+#endif
+#endif
+
+#define PERTHREAD __thread
+
+#endif
+
+#ifndef _darwin_
+
+#ifndef Y_ARRAY_SIZE
+#define Y_ARRAY_SIZE(a) (sizeof(a) / sizeof((a)[0]))
+#endif
+
+#ifndef NDEBUG
+#define DBG_FILL_MEMORY
+static bool FillMemoryOnAllocation = true;
+#endif
+
+static bool TransparentHugePages = false; // force MADV_HUGEPAGE for large allocs
+static bool MapHugeTLB = false; // force MAP_HUGETLB for small allocs
+static bool EnableDefrag = true;
+
+// Buffers that are larger than this size will not be filled with 0xcf
+#ifndef DBG_FILL_MAX_SIZE
+#define DBG_FILL_MAX_SIZE 0x01000000000000ULL
+#endif
+
+template
+inline T* DoCas(T* volatile* target, T* exchange, T* compare) {
+#if defined(_linux_)
+ return __sync_val_compare_and_swap(target, compare, exchange);
+#elif defined(_WIN32)
+#ifdef _64_
+ return (T*)_InterlockedCompareExchange64((__int64*)target, (__int64)exchange, (__int64)compare);
+#else
+ //return (T*)InterlockedCompareExchangePointer(targetVoidP, exchange, compare);
+ return (T*)_InterlockedCompareExchange((LONG*)target, (LONG)exchange, (LONG)compare);
+#endif
+#elif defined(__i386) || defined(__x86_64__)
+ union {
+ T* volatile* NP;
+ void* volatile* VoidP;
+ } gccSucks;
+ gccSucks.NP = target;
+ void* volatile* targetVoidP = gccSucks.VoidP;
+
+ __asm__ __volatile__(
+ "lock\n\t"
+ "cmpxchg %2,%0\n\t"
+ : "+m"(*(targetVoidP)), "+a"(compare)
+ : "r"(exchange)
+ : "cc", "memory");
+ return compare;
+#else
+#error inline_cas not defined for this platform
+#endif
+}
+
+#ifdef _64_
+const uintptr_t N_MAX_WORKSET_SIZE = 0x100000000ll * 200;
+const uintptr_t N_HUGE_AREA_FINISH = 0x700000000000ll;
+#ifndef _freebsd_
+const uintptr_t LINUX_MMAP_AREA_START = 0x100000000ll;
+static uintptr_t volatile linuxAllocPointer = LINUX_MMAP_AREA_START;
+static uintptr_t volatile linuxAllocPointerHuge = LINUX_MMAP_AREA_START + N_MAX_WORKSET_SIZE;
+#endif
+#else
+const uintptr_t N_MAX_WORKSET_SIZE = 0xffffffff;
+#endif
+#define ALLOC_START ((char*)0)
+
+const size_t N_CHUNK_SIZE = 1024 * 1024;
+const size_t N_CHUNKS = N_MAX_WORKSET_SIZE / N_CHUNK_SIZE;
+const size_t N_LARGE_ALLOC_SIZE = N_CHUNK_SIZE * 128;
+
+// map size idx to size in bytes
+#ifdef LFALLOC_YT
+const int N_SIZES = 27;
+#else
+const int N_SIZES = 25;
+#endif
+const int nSizeIdxToSize[N_SIZES] = {
+ -1,
+#if defined(_64_)
+ 16, 16, 32, 32, 48, 64, 96, 128,
+#else
+ 8,
+ 16,
+ 24,
+ 32,
+ 48,
+ 64,
+ 96,
+ 128,
+#endif
+ 192, 256, 384, 512, 768, 1024, 1536, 2048,
+ 3072, 4096, 6144, 8192, 12288, 16384, 24576, 32768,
+#ifdef LFALLOC_YT
+ 49152, 65536
+#endif
+};
+#ifdef LFALLOC_YT
+const size_t N_MAX_FAST_SIZE = 65536;
+#else
+const size_t N_MAX_FAST_SIZE = 32768;
+#endif
+const unsigned char size2idxArr1[64 + 1] = {
+ 1,
+#if defined(_64_)
+ 2, 2, 4, 4, // 16, 16, 32, 32
+#else
+ 1, 2, 3, 4, // 8, 16, 24, 32
+#endif
+ 5, 5, 6, 6, // 48, 64
+ 7, 7, 7, 7, 8, 8, 8, 8, // 96, 128
+ 9, 9, 9, 9, 9, 9, 9, 9, 10, 10, 10, 10, 10, 10, 10, 10, // 192, 256
+ 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, // 384
+ 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12 // 512
+};
+#ifdef LFALLOC_YT
+const unsigned char size2idxArr2[256] = {
+#else
+const unsigned char size2idxArr2[128] = {
+#endif
+ 12, 12, 13, 14, // 512, 512, 768, 1024
+ 15, 15, 16, 16, // 1536, 2048
+ 17, 17, 17, 17, 18, 18, 18, 18, // 3072, 4096
+ 19, 19, 19, 19, 19, 19, 19, 19, 20, 20, 20, 20, 20, 20, 20, 20, // 6144, 8192
+ 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, // 12288
+ 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, // 16384
+ 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23,
+ 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, 23, // 24576
+ 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24,
+ 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, // 32768
+#ifdef LFALLOC_YT
+ 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25,
+ 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25,
+ 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25,
+ 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, // 49152
+ 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26,
+ 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26,
+ 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26,
+ 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, // 65536
+#endif
+};
+
+// map entry number to size idx
+// special size idx's: 0 = not used, -1 = mem locked, but not allocated
+static volatile char chunkSizeIdx[N_CHUNKS];
+const int FREE_CHUNK_ARR_BUF = 0x20000; // this is effectively 128G of free memory (with 1M chunks), should not be exhausted actually
+static volatile uintptr_t freeChunkArr[FREE_CHUNK_ARR_BUF];
+static volatile int freeChunkCount;
+
+static void AddFreeChunk(uintptr_t chunkId) {
+ chunkSizeIdx[chunkId] = -1;
+ if (Y_UNLIKELY(freeChunkCount == FREE_CHUNK_ARR_BUF))
+ NMalloc::AbortFromCorruptedAllocator(); // free chunks arrray overflowed
+ freeChunkArr[freeChunkCount++] = chunkId;
+}
+
+static bool GetFreeChunk(uintptr_t* res) {
+ if (freeChunkCount == 0) {
+ *res = 0;
+ return false;
+ }
+ *res = freeChunkArr[--freeChunkCount];
+ return true;
+}
+
+//////////////////////////////////////////////////////////////////////////
+enum ELFAllocCounter {
+ CT_USER_ALLOC, // accumulated size requested by user code
+ CT_MMAP, // accumulated mmapped size
+ CT_MMAP_CNT, // number of mmapped regions
+ CT_MUNMAP, // accumulated unmmapped size
+ CT_MUNMAP_CNT, // number of munmaped regions
+ CT_SYSTEM_ALLOC, // accumulated allocated size for internal lfalloc needs
+ CT_SYSTEM_FREE, // accumulated deallocated size for internal lfalloc needs
+ CT_SMALL_ALLOC, // accumulated allocated size for fixed-size blocks
+ CT_SMALL_FREE, // accumulated deallocated size for fixed-size blocks
+ CT_LARGE_ALLOC, // accumulated allocated size for large blocks
+ CT_LARGE_FREE, // accumulated deallocated size for large blocks
+ CT_SLOW_ALLOC_CNT, // number of slow (not LF) allocations
+ CT_DEGRAGMENT_CNT, // number of memory defragmentations
+ CT_MAX
+};
+
+static Y_FORCE_INLINE void IncrementCounter(ELFAllocCounter counter, size_t value);
+
+//////////////////////////////////////////////////////////////////////////
+enum EMMapMode {
+ MM_NORMAL, // memory for small allocs
+ MM_HUGE // memory for large allocs
+};
+
+#ifndef _MSC_VER
+inline void VerifyMmapResult(void* result) {
+ if (Y_UNLIKELY(result == MAP_FAILED))
+ NMalloc::AbortFromCorruptedAllocator(); // negative size requested? or just out of mem
+}
+#endif
+
+#if !defined(_MSC_VER) && !defined(_freebsd_) && defined(_64_)
+static char* AllocWithMMapLinuxImpl(uintptr_t sz, EMMapMode mode) {
+ char* volatile* areaPtr;
+ char* areaStart;
+ uintptr_t areaFinish;
+
+ int mapProt = PROT_READ | PROT_WRITE;
+ int mapFlags = MAP_PRIVATE | MAP_ANON;
+
+ if (mode == MM_HUGE) {
+ areaPtr = reinterpret_cast(&linuxAllocPointerHuge);
+ areaStart = reinterpret_cast(LINUX_MMAP_AREA_START + N_MAX_WORKSET_SIZE);
+ areaFinish = N_HUGE_AREA_FINISH;
+ } else {
+ areaPtr = reinterpret_cast(&linuxAllocPointer);
+ areaStart = reinterpret_cast(LINUX_MMAP_AREA_START);
+ areaFinish = N_MAX_WORKSET_SIZE;
+
+ if (MapHugeTLB) {
+ mapFlags |= MAP_HUGETLB;
+ }
+ }
+
+ bool wrapped = false;
+ for (;;) {
+ char* prevAllocPtr = *areaPtr;
+ char* nextAllocPtr = prevAllocPtr + sz;
+ if (uintptr_t(nextAllocPtr - (char*)nullptr) >= areaFinish) {
+ if (Y_UNLIKELY(wrapped)) {
+ // virtual memory is over fragmented
+ NMalloc::AbortFromCorruptedAllocator();
+ }
+ // wrap after all area is used
+ DoCas(areaPtr, areaStart, prevAllocPtr);
+ wrapped = true;
+ continue;
+ }
+
+ if (DoCas(areaPtr, nextAllocPtr, prevAllocPtr) != prevAllocPtr)
+ continue;
+
+ char* largeBlock = (char*)mmap(prevAllocPtr, sz, mapProt, mapFlags, -1, 0);
+ VerifyMmapResult(largeBlock);
+ if (largeBlock == prevAllocPtr)
+ return largeBlock;
+ if (largeBlock)
+ munmap(largeBlock, sz);
+
+ if (sz < 0x80000) {
+ // skip utilized area with big steps
+ DoCas(areaPtr, nextAllocPtr + 0x10 * 0x10000, nextAllocPtr);
+ }
+ }
+}
+#endif
+
+static char* AllocWithMMap(uintptr_t sz, EMMapMode mode) {
+ (void)mode;
+#ifdef _MSC_VER
+ char* largeBlock = (char*)VirtualAlloc(0, sz, MEM_RESERVE, PAGE_READWRITE);
+ if (Y_UNLIKELY(largeBlock == nullptr))
+ NMalloc::AbortFromCorruptedAllocator(); // out of memory
+ if (Y_UNLIKELY(uintptr_t(((char*)largeBlock - ALLOC_START) + sz) >= N_MAX_WORKSET_SIZE))
+ NMalloc::AbortFromCorruptedAllocator(); // out of working set, something has broken
+#else
+#if defined(_freebsd_) || !defined(_64_) || defined(USE_LFALLOC_RANDOM_HINT)
+ uintptr_t areaStart;
+ uintptr_t areaFinish;
+ if (mode == MM_HUGE) {
+ areaStart = LINUX_MMAP_AREA_START + N_MAX_WORKSET_SIZE;
+ areaFinish = N_HUGE_AREA_FINISH;
+ } else {
+ areaStart = LINUX_MMAP_AREA_START;
+ areaFinish = N_MAX_WORKSET_SIZE;
+ }
+#if defined(USE_LFALLOC_RANDOM_HINT)
+ static thread_local std::mt19937_64 generator(std::random_device{}());
+ std::uniform_int_distribution distr(areaStart, areaFinish / 2);
+ char* largeBlock = (char*)mmap(reinterpret_cast(distr(generator)), sz, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+#else
+ char* largeBlock = (char*)mmap(0, sz, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0);
+#endif
+ VerifyMmapResult(largeBlock);
+ if (Y_UNLIKELY(uintptr_t(((char*)largeBlock - ALLOC_START) + sz) >= areaFinish))
+ NMalloc::AbortFromCorruptedAllocator(); // out of working set, something has broken
+#else
+ char* largeBlock = AllocWithMMapLinuxImpl(sz, mode);
+ if (TransparentHugePages) {
+ madvise(largeBlock, sz, MADV_HUGEPAGE);
+ }
+#endif
+#endif
+ Y_ASSERT_NOBT(largeBlock);
+ IncrementCounter(CT_MMAP, sz);
+ IncrementCounter(CT_MMAP_CNT, 1);
+ return largeBlock;
+}
+
+enum class ELarge : ui8 {
+ Free = 0, // block in free cache
+ Alloc = 1, // block is allocated
+ Gone = 2, // block was unmapped
+};
+
+struct TLargeBlk {
+
+ static TLargeBlk* As(void *raw) {
+ return reinterpret_cast((char*)raw - 4096ll);
+ }
+
+ static const TLargeBlk* As(const void *raw) {
+ return reinterpret_cast((const char*)raw - 4096ll);
+ }
+
+ void SetSize(size_t bytes, size_t pages) {
+ Pages = pages;
+ Bytes = bytes;
+ }
+
+ void Mark(ELarge state) {
+ const ui64 marks[] = {
+ 0x8b38aa5ca4953c98, // ELarge::Free
+ 0xf916d33584eb5087, // ELarge::Alloc
+ 0xd33b0eca7651bc3f // ELarge::Gone
+ };
+
+ Token = size_t(marks[ui8(state)]);
+ }
+
+ size_t Pages; // Total pages allocated with mmap like call
+ size_t Bytes; // Actually requested bytes by user
+ size_t Token; // Block state token, see ELarge enum.
+};
+
+
+static void LargeBlockUnmap(void* p, size_t pages) {
+ const auto bytes = (pages + 1) * uintptr_t(4096);
+
+ IncrementCounter(CT_MUNMAP, bytes);
+ IncrementCounter(CT_MUNMAP_CNT, 1);
+#ifdef _MSC_VER
+ Y_ASSERT_NOBT(0);
+#else
+ TLargeBlk::As(p)->Mark(ELarge::Gone);
+ munmap((char*)p - 4096ll, bytes);
+#endif
+}
+
+//////////////////////////////////////////////////////////////////////////
+const size_t LB_BUF_SIZE = 250;
+const size_t LB_BUF_HASH = 977;
+static int LB_LIMIT_TOTAL_SIZE = 500 * 1024 * 1024 / 4096; // do not keep more then this mem total in lbFreePtrs[]
+static void* volatile lbFreePtrs[LB_BUF_HASH][LB_BUF_SIZE];
+static TAtomic lbFreePageCount;
+
+
+static void* LargeBlockAlloc(size_t _nSize, ELFAllocCounter counter) {
+ size_t pgCount = (_nSize + 4095) / 4096;
+#ifdef _MSC_VER
+ char* pRes = (char*)VirtualAlloc(0, (pgCount + 1) * 4096ll, MEM_COMMIT, PAGE_READWRITE);
+ if (Y_UNLIKELY(pRes == 0)) {
+ NMalloc::AbortFromCorruptedAllocator(); // out of memory
+ }
+#else
+
+ IncrementCounter(counter, pgCount * 4096ll);
+ IncrementCounter(CT_SYSTEM_ALLOC, 4096ll);
+
+ int lbHash = pgCount % LB_BUF_HASH;
+ for (int i = 0; i < LB_BUF_SIZE; ++i) {
+ void* p = lbFreePtrs[lbHash][i];
+ if (p == nullptr)
+ continue;
+ if (DoCas(&lbFreePtrs[lbHash][i], (void*)nullptr, p) == p) {
+ size_t realPageCount = TLargeBlk::As(p)->Pages;
+ if (realPageCount == pgCount) {
+ AtomicAdd(lbFreePageCount, -pgCount);
+ TLargeBlk::As(p)->Mark(ELarge::Alloc);
+ return p;
+ } else {
+ if (DoCas(&lbFreePtrs[lbHash][i], p, (void*)nullptr) != (void*)nullptr) {
+ // block was freed while we were busy
+ AtomicAdd(lbFreePageCount, -realPageCount);
+ LargeBlockUnmap(p, realPageCount);
+ --i;
+ }
+ }
+ }
+ }
+ char* pRes = AllocWithMMap((pgCount + 1) * 4096ll, MM_HUGE);
+#endif
+ pRes += 4096ll;
+ TLargeBlk::As(pRes)->SetSize(_nSize, pgCount);
+ TLargeBlk::As(pRes)->Mark(ELarge::Alloc);
+
+ return pRes;
+}
+
+#ifndef _MSC_VER
+static void FreeAllLargeBlockMem() {
+ for (auto& lbFreePtr : lbFreePtrs) {
+ for (int i = 0; i < LB_BUF_SIZE; ++i) {
+ void* p = lbFreePtr[i];
+ if (p == nullptr)
+ continue;
+ if (DoCas(&lbFreePtr[i], (void*)nullptr, p) == p) {
+ int pgCount = TLargeBlk::As(p)->Pages;
+ AtomicAdd(lbFreePageCount, -pgCount);
+ LargeBlockUnmap(p, pgCount);
+ }
+ }
+ }
+}
+#endif
+
+static void LargeBlockFree(void* p, ELFAllocCounter counter) {
+ if (p == nullptr)
+ return;
+#ifdef _MSC_VER
+ VirtualFree((char*)p - 4096ll, 0, MEM_RELEASE);
+#else
+ size_t pgCount = TLargeBlk::As(p)->Pages;
+
+ TLargeBlk::As(p)->Mark(ELarge::Free);
+ IncrementCounter(counter, pgCount * 4096ll);
+ IncrementCounter(CT_SYSTEM_FREE, 4096ll);
+
+ if (lbFreePageCount > LB_LIMIT_TOTAL_SIZE)
+ FreeAllLargeBlockMem();
+ int lbHash = pgCount % LB_BUF_HASH;
+ for (int i = 0; i < LB_BUF_SIZE; ++i) {
+ if (lbFreePtrs[lbHash][i] == nullptr) {
+ if (DoCas(&lbFreePtrs[lbHash][i], p, (void*)nullptr) == nullptr) {
+ AtomicAdd(lbFreePageCount, pgCount);
+ return;
+ }
+ }
+ }
+
+ LargeBlockUnmap(p, pgCount);
+#endif
+}
+
+static void* SystemAlloc(size_t _nSize) {
+ //HeapAlloc(GetProcessHeap(), HEAP_GENERATE_EXCEPTIONS, _nSize);
+ return LargeBlockAlloc(_nSize, CT_SYSTEM_ALLOC);
+}
+static void SystemFree(void* p) {
+ //HeapFree(GetProcessHeap(), 0, p);
+ LargeBlockFree(p, CT_SYSTEM_FREE);
+}
+
+//////////////////////////////////////////////////////////////////////////
+static int* volatile nLock = nullptr;
+static int nLockVar;
+inline void RealEnterCriticalDefault(int* volatile* lockPtr) {
+ while (DoCas(lockPtr, &nLockVar, (int*)nullptr) != nullptr)
+ ; //pthread_yield();
+}
+inline void RealLeaveCriticalDefault(int* volatile* lockPtr) {
+ *lockPtr = nullptr;
+}
+static void (*RealEnterCritical)(int* volatile* lockPtr) = RealEnterCriticalDefault;
+static void (*RealLeaveCritical)(int* volatile* lockPtr) = RealLeaveCriticalDefault;
+static void (*BeforeLFAllocGlobalLockAcquired)() = nullptr;
+static void (*AfterLFAllocGlobalLockReleased)() = nullptr;
+class CCriticalSectionLockMMgr {
+public:
+ CCriticalSectionLockMMgr() {
+ if (BeforeLFAllocGlobalLockAcquired) {
+ BeforeLFAllocGlobalLockAcquired();
+ }
+ RealEnterCritical(&nLock);
+ }
+ ~CCriticalSectionLockMMgr() {
+ RealLeaveCritical(&nLock);
+ if (AfterLFAllocGlobalLockReleased) {
+ AfterLFAllocGlobalLockReleased();
+ }
+ }
+};
+
+//////////////////////////////////////////////////////////////////////////
+class TLFAllocFreeList {
+ struct TNode {
+ TNode* Next;
+ };
+
+ TNode* volatile Head;
+ TNode* volatile Pending;
+ TAtomic PendingToFreeListCounter;
+ TAtomic AllocCount;
+
+ static Y_FORCE_INLINE void Enqueue(TNode* volatile* headPtr, TNode* n) {
+ for (;;) {
+ TNode* volatile prevHead = *headPtr;
+ n->Next = prevHead;
+ if (DoCas(headPtr, n, prevHead) == prevHead)
+ break;
+ }
+ }
+ Y_FORCE_INLINE void* DoAlloc() {
+ TNode* res;
+ for (res = Head; res; res = Head) {
+ TNode* keepNext = res->Next;
+ if (DoCas(&Head, keepNext, res) == res) {
+ //Y_VERIFY(keepNext == res->Next);
+ break;
+ }
+ }
+ return res;
+ }
+ void FreeList(TNode* fl) {
+ if (!fl)
+ return;
+ TNode* flTail = fl;
+ while (flTail->Next)
+ flTail = flTail->Next;
+ for (;;) {
+ TNode* volatile prevHead = Head;
+ flTail->Next = prevHead;
+ if (DoCas(&Head, fl, prevHead) == prevHead)
+ break;
+ }
+ }
+
+public:
+ Y_FORCE_INLINE void Free(void* ptr) {
+ TNode* newFree = (TNode*)ptr;
+ if (AtomicAdd(AllocCount, 0) == 0)
+ Enqueue(&Head, newFree);
+ else
+ Enqueue(&Pending, newFree);
+ }
+ Y_FORCE_INLINE void* Alloc() {
+ TAtomic keepCounter = AtomicAdd(PendingToFreeListCounter, 0);
+ TNode* fl = Pending;
+ if (AtomicAdd(AllocCount, 1) == 1) {
+ // No other allocs in progress.
+ // If (keepCounter == PendingToFreeListCounter) then Pending was not freed by other threads.
+ // Hence Pending is not used in any concurrent DoAlloc() atm and can be safely moved to FreeList
+ if (fl && keepCounter == AtomicAdd(PendingToFreeListCounter, 0) && DoCas(&Pending, (TNode*)nullptr, fl) == fl) {
+ // pick first element from Pending and return it
+ void* res = fl;
+ fl = fl->Next;
+ // if there are other elements in Pending list, add them to main free list
+ FreeList(fl);
+ AtomicAdd(PendingToFreeListCounter, 1);
+ AtomicAdd(AllocCount, -1);
+ return res;
+ }
+ }
+ void* res = DoAlloc();
+ AtomicAdd(AllocCount, -1);
+ return res;
+ }
+ void* GetWholeList() {
+ TNode* res;
+ for (res = Head; res; res = Head) {
+ if (DoCas(&Head, (TNode*)nullptr, res) == res)
+ break;
+ }
+ return res;
+ }
+ void ReturnWholeList(void* ptr) {
+ while (AtomicAdd(AllocCount, 0) != 0) // theoretically can run into problems with parallel DoAlloc()
+ ; //ThreadYield();
+ for (;;) {
+ TNode* prevHead = Head;
+ if (DoCas(&Head, (TNode*)ptr, prevHead) == prevHead) {
+ FreeList(prevHead);
+ break;
+ }
+ }
+ }
+};
+
+/////////////////////////////////////////////////////////////////////////
+static TLFAllocFreeList globalFreeLists[N_SIZES];
+static char* volatile globalCurrentPtr[N_SIZES];
+static TLFAllocFreeList blockFreeList;
+
+// globalFreeLists[] contains TFreeListGroup, each of them points up to 15 free blocks
+const int FL_GROUP_SIZE = 15;
+struct TFreeListGroup {
+ TFreeListGroup* Next;
+ char* Ptrs[FL_GROUP_SIZE];
+};
+#ifdef _64_
+const int FREE_LIST_GROUP_SIZEIDX = 8;
+#else
+const int FREE_LIST_GROUP_SIZEIDX = 6;
+#endif
+
+//////////////////////////////////////////////////////////////////////////
+// find free chunks and reset chunk size so they can be reused by different sized allocations
+// do not look at blockFreeList (TFreeListGroup has same size for any allocations)
+static bool DefragmentMem() {
+ if (!EnableDefrag) {
+ return false;
+ }
+
+ IncrementCounter(CT_DEGRAGMENT_CNT, 1);
+
+ int* nFreeCount = (int*)SystemAlloc(N_CHUNKS * sizeof(int));
+ if (Y_UNLIKELY(!nFreeCount)) {
+ //__debugbreak();
+ NMalloc::AbortFromCorruptedAllocator();
+ }
+ memset(nFreeCount, 0, N_CHUNKS * sizeof(int));
+
+ TFreeListGroup* wholeLists[N_SIZES];
+ for (int nSizeIdx = 0; nSizeIdx < N_SIZES; ++nSizeIdx) {
+ wholeLists[nSizeIdx] = (TFreeListGroup*)globalFreeLists[nSizeIdx].GetWholeList();
+ for (TFreeListGroup* g = wholeLists[nSizeIdx]; g; g = g->Next) {
+ for (auto pData : g->Ptrs) {
+ if (pData) {
+ uintptr_t nChunk = (pData - ALLOC_START) / N_CHUNK_SIZE;
+ ++nFreeCount[nChunk];
+ Y_ASSERT_NOBT(chunkSizeIdx[nChunk] == nSizeIdx);
+ }
+ }
+ }
+ }
+
+ bool bRes = false;
+ for (size_t nChunk = 0; nChunk < N_CHUNKS; ++nChunk) {
+ int fc = nFreeCount[nChunk];
+ nFreeCount[nChunk] = 0;
+ if (chunkSizeIdx[nChunk] <= 0)
+ continue;
+ int nEntries = N_CHUNK_SIZE / nSizeIdxToSize[static_cast(chunkSizeIdx[nChunk])];
+ Y_ASSERT_NOBT(fc <= nEntries); // can not have more free blocks then total count
+ if (fc == nEntries) {
+ bRes = true;
+ nFreeCount[nChunk] = 1;
+ }
+ }
+ if (bRes) {
+ for (auto& wholeList : wholeLists) {
+ TFreeListGroup** ppPtr = &wholeList;
+ while (*ppPtr) {
+ TFreeListGroup* g = *ppPtr;
+ int dst = 0;
+ for (auto pData : g->Ptrs) {
+ if (pData) {
+ uintptr_t nChunk = (pData - ALLOC_START) / N_CHUNK_SIZE;
+ if (nFreeCount[nChunk] == 0)
+ g->Ptrs[dst++] = pData; // block is not freed, keep pointer
+ }
+ }
+ if (dst == 0) {
+ // no valid pointers in group, free it
+ *ppPtr = g->Next;
+ blockFreeList.Free(g);
+ } else {
+ // reset invalid pointers to 0
+ for (int i = dst; i < FL_GROUP_SIZE; ++i)
+ g->Ptrs[i] = nullptr;
+ ppPtr = &g->Next;
+ }
+ }
+ }
+ for (uintptr_t nChunk = 0; nChunk < N_CHUNKS; ++nChunk) {
+ if (!nFreeCount[nChunk])
+ continue;
+ char* pStart = ALLOC_START + nChunk * N_CHUNK_SIZE;
+#ifdef _win_
+ VirtualFree(pStart, N_CHUNK_SIZE, MEM_DECOMMIT);
+#elif defined(_freebsd_)
+ madvise(pStart, N_CHUNK_SIZE, MADV_FREE);
+#else
+ madvise(pStart, N_CHUNK_SIZE, MADV_DONTNEED);
+#endif
+ AddFreeChunk(nChunk);
+ }
+ }
+
+ for (int nSizeIdx = 0; nSizeIdx < N_SIZES; ++nSizeIdx)
+ globalFreeLists[nSizeIdx].ReturnWholeList(wholeLists[nSizeIdx]);
+
+ SystemFree(nFreeCount);
+ return bRes;
+}
+
+static Y_FORCE_INLINE void* LFAllocFromCurrentChunk(int nSizeIdx, int blockSize, int count) {
+ char* volatile* pFreeArray = &globalCurrentPtr[nSizeIdx];
+ while (char* newBlock = *pFreeArray) {
+ char* nextFree = newBlock + blockSize * count;
+
+ // check if there is space in chunk
+ char* globalEndPtr = ALLOC_START + ((newBlock - ALLOC_START) & ~((uintptr_t)N_CHUNK_SIZE - 1)) + N_CHUNK_SIZE;
+ if (nextFree >= globalEndPtr) {
+ if (nextFree > globalEndPtr)
+ break;
+ nextFree = nullptr; // it was last block in chunk
+ }
+ if (DoCas(pFreeArray, nextFree, newBlock) == newBlock)
+ return newBlock;
+ }
+ return nullptr;
+}
+
+enum EDefrag {
+ MEM_DEFRAG,
+ NO_MEM_DEFRAG,
+};
+
+static void* SlowLFAlloc(int nSizeIdx, int blockSize, EDefrag defrag) {
+ IncrementCounter(CT_SLOW_ALLOC_CNT, 1);
+
+ CCriticalSectionLockMMgr ls;
+ void* res = LFAllocFromCurrentChunk(nSizeIdx, blockSize, 1);
+ if (res)
+ return res; // might happen when other thread allocated new current chunk
+
+ for (;;) {
+ uintptr_t nChunk;
+ if (GetFreeChunk(&nChunk)) {
+ char* newPlace = ALLOC_START + nChunk * N_CHUNK_SIZE;
+#ifdef _MSC_VER
+ void* pTest = VirtualAlloc(newPlace, N_CHUNK_SIZE, MEM_COMMIT, PAGE_READWRITE);
+ Y_ASSERT_NOBT(pTest == newPlace);
+#endif
+ chunkSizeIdx[nChunk] = (char)nSizeIdx;
+ globalCurrentPtr[nSizeIdx] = newPlace + blockSize;
+ return newPlace;
+ }
+
+ // out of luck, try to defrag
+ if (defrag == MEM_DEFRAG && DefragmentMem()) {
+ continue;
+ }
+
+ char* largeBlock = AllocWithMMap(N_LARGE_ALLOC_SIZE, MM_NORMAL);
+ uintptr_t addr = ((largeBlock - ALLOC_START) + N_CHUNK_SIZE - 1) & (~(N_CHUNK_SIZE - 1));
+ uintptr_t endAddr = ((largeBlock - ALLOC_START) + N_LARGE_ALLOC_SIZE) & (~(N_CHUNK_SIZE - 1));
+ for (uintptr_t p = addr; p < endAddr; p += N_CHUNK_SIZE) {
+ uintptr_t chunk = p / N_CHUNK_SIZE;
+ Y_ASSERT_NOBT(chunk * N_CHUNK_SIZE == p);
+ Y_ASSERT_NOBT(chunkSizeIdx[chunk] == 0);
+ AddFreeChunk(chunk);
+ }
+ }
+ return nullptr;
+}
+
+// allocate single block
+static Y_FORCE_INLINE void* LFAllocNoCache(int nSizeIdx, EDefrag defrag) {
+ int blockSize = nSizeIdxToSize[nSizeIdx];
+ void* res = LFAllocFromCurrentChunk(nSizeIdx, blockSize, 1);
+ if (res)
+ return res;
+
+ return SlowLFAlloc(nSizeIdx, blockSize, defrag);
+}
+
+// allocate multiple blocks, returns number of blocks allocated (max FL_GROUP_SIZE)
+// buf should have space for at least FL_GROUP_SIZE elems
+static Y_FORCE_INLINE int LFAllocNoCacheMultiple(int nSizeIdx, char** buf) {
+ int blockSize = nSizeIdxToSize[nSizeIdx];
+ void* res = LFAllocFromCurrentChunk(nSizeIdx, blockSize, FL_GROUP_SIZE);
+ if (res) {
+ char* resPtr = (char*)res;
+ for (int k = 0; k < FL_GROUP_SIZE; ++k) {
+ buf[k] = resPtr;
+ resPtr += blockSize;
+ }
+ return FL_GROUP_SIZE;
+ }
+ buf[0] = (char*)SlowLFAlloc(nSizeIdx, blockSize, MEM_DEFRAG);
+ return 1;
+}
+
+// take several blocks from global free list (max FL_GROUP_SIZE blocks), returns number of blocks taken
+// buf should have space for at least FL_GROUP_SIZE elems
+static Y_FORCE_INLINE int TakeBlocksFromGlobalFreeList(int nSizeIdx, char** buf) {
+ TLFAllocFreeList& fl = globalFreeLists[nSizeIdx];
+ TFreeListGroup* g = (TFreeListGroup*)fl.Alloc();
+ if (g) {
+ int resCount = 0;
+ for (auto& ptr : g->Ptrs) {
+ if (ptr)
+ buf[resCount++] = ptr;
+ else
+ break;
+ }
+ blockFreeList.Free(g);
+ return resCount;
+ }
+ return 0;
+}
+
+// add several blocks to global free list
+static Y_FORCE_INLINE void PutBlocksToGlobalFreeList(ptrdiff_t nSizeIdx, char** buf, int count) {
+ for (int startIdx = 0; startIdx < count;) {
+ TFreeListGroup* g = (TFreeListGroup*)blockFreeList.Alloc();
+ Y_ASSERT_NOBT(sizeof(TFreeListGroup) == nSizeIdxToSize[FREE_LIST_GROUP_SIZEIDX]);
+ if (!g) {
+ g = (TFreeListGroup*)LFAllocNoCache(FREE_LIST_GROUP_SIZEIDX, NO_MEM_DEFRAG);
+ }
+
+ int groupSize = count - startIdx;
+ if (groupSize > FL_GROUP_SIZE)
+ groupSize = FL_GROUP_SIZE;
+ for (int i = 0; i < groupSize; ++i)
+ g->Ptrs[i] = buf[startIdx + i];
+ for (int i = groupSize; i < FL_GROUP_SIZE; ++i)
+ g->Ptrs[i] = nullptr;
+
+ // add free group to the global list
+ TLFAllocFreeList& fl = globalFreeLists[nSizeIdx];
+ fl.Free(g);
+
+ startIdx += groupSize;
+ }
+}
+
+//////////////////////////////////////////////////////////////////////////
+static TAtomic GlobalCounters[CT_MAX];
+const int MAX_LOCAL_UPDATES = 100;
+
+struct TLocalCounter {
+ intptr_t Value;
+ int Updates;
+ TAtomic* Parent;
+
+ Y_FORCE_INLINE void Init(TAtomic* parent) {
+ Parent = parent;
+ Value = 0;
+ Updates = 0;
+ }
+
+ Y_FORCE_INLINE void Increment(size_t value) {
+ Value += value;
+ if (++Updates > MAX_LOCAL_UPDATES) {
+ Flush();
+ }
+ }
+
+ Y_FORCE_INLINE void Flush() {
+ AtomicAdd(*Parent, Value);
+ Value = 0;
+ Updates = 0;
+ }
+};
+
+////////////////////////////////////////////////////////////////////////////////
+// DBG stuff
+////////////////////////////////////////////////////////////////////////////////
+
+#if defined(LFALLOC_DBG)
+
+struct TPerTagAllocCounter {
+ TAtomic Size;
+ TAtomic Count;
+
+ Y_FORCE_INLINE void Alloc(size_t size) {
+ AtomicAdd(Size, size);
+ AtomicAdd(Count, 1);
+ }
+
+ Y_FORCE_INLINE void Free(size_t size) {
+ AtomicSub(Size, size);
+ AtomicSub(Count, 1);
+ }
+};
+
+struct TLocalPerTagAllocCounter {
+ intptr_t Size;
+ int Count;
+ int Updates;
+
+ Y_FORCE_INLINE void Init() {
+ Size = 0;
+ Count = 0;
+ Updates = 0;
+ }
+
+ Y_FORCE_INLINE void Alloc(TPerTagAllocCounter& parent, size_t size) {
+ Size += size;
+ ++Count;
+ if (++Updates > MAX_LOCAL_UPDATES) {
+ Flush(parent);
+ }
+ }
+
+ Y_FORCE_INLINE void Free(TPerTagAllocCounter& parent, size_t size) {
+ Size -= size;
+ --Count;
+ if (++Updates > MAX_LOCAL_UPDATES) {
+ Flush(parent);
+ }
+ }
+
+ Y_FORCE_INLINE void Flush(TPerTagAllocCounter& parent) {
+ AtomicAdd(parent.Size, Size);
+ Size = 0;
+ AtomicAdd(parent.Count, Count);
+ Count = 0;
+ Updates = 0;
+ }
+};
+
+static const int DBG_ALLOC_MAX_TAG = 1000;
+static const int DBG_ALLOC_NUM_SIZES = 30;
+static TPerTagAllocCounter GlobalPerTagAllocCounters[DBG_ALLOC_MAX_TAG][DBG_ALLOC_NUM_SIZES];
+
+#endif // LFALLOC_DBG
+
+//////////////////////////////////////////////////////////////////////////
+const int THREAD_BUF = 256;
+static int borderSizes[N_SIZES];
+const int MAX_MEM_PER_SIZE_PER_THREAD = 512 * 1024;
+struct TThreadAllocInfo {
+ // FreePtrs - pointers to first free blocks in per thread block list
+ // LastFreePtrs - pointers to last blocks in lists, may be invalid if FreePtr is zero
+ char* FreePtrs[N_SIZES][THREAD_BUF];
+ int FreePtrIndex[N_SIZES];
+ TThreadAllocInfo* pNextInfo;
+ TLocalCounter LocalCounters[CT_MAX];
+
+#if defined(LFALLOC_DBG)
+ TLocalPerTagAllocCounter LocalPerTagAllocCounters[DBG_ALLOC_MAX_TAG][DBG_ALLOC_NUM_SIZES];
+#endif
+#ifdef _win_
+ HANDLE hThread;
+#endif
+
+ void Init(TThreadAllocInfo** pHead) {
+ memset(this, 0, sizeof(*this));
+ for (auto& i : FreePtrIndex)
+ i = THREAD_BUF;
+#ifdef _win_
+ BOOL b = DuplicateHandle(
+ GetCurrentProcess(), GetCurrentThread(),
+ GetCurrentProcess(), &hThread,
+ 0, FALSE, DUPLICATE_SAME_ACCESS);
+ Y_ASSERT_NOBT(b);
+#endif
+ pNextInfo = *pHead;
+ *pHead = this;
+ for (int k = 0; k < N_SIZES; ++k) {
+ int maxCount = MAX_MEM_PER_SIZE_PER_THREAD / nSizeIdxToSize[k];
+ if (maxCount > THREAD_BUF)
+ maxCount = THREAD_BUF;
+ borderSizes[k] = THREAD_BUF - maxCount;
+ }
+ for (int i = 0; i < CT_MAX; ++i) {
+ LocalCounters[i].Init(&GlobalCounters[i]);
+ }
+#if defined(LFALLOC_DBG)
+ for (int tag = 0; tag < DBG_ALLOC_MAX_TAG; ++tag) {
+ for (int sizeIdx = 0; sizeIdx < DBG_ALLOC_NUM_SIZES; ++sizeIdx) {
+ auto& local = LocalPerTagAllocCounters[tag][sizeIdx];
+ local.Init();
+ }
+ }
+#endif
+ }
+ void Done() {
+ for (auto sizeIdx : FreePtrIndex) {
+ Y_ASSERT_NOBT(sizeIdx == THREAD_BUF);
+ }
+ for (auto& localCounter : LocalCounters) {
+ localCounter.Flush();
+ }
+#if defined(LFALLOC_DBG)
+ for (int tag = 0; tag < DBG_ALLOC_MAX_TAG; ++tag) {
+ for (int sizeIdx = 0; sizeIdx < DBG_ALLOC_NUM_SIZES; ++sizeIdx) {
+ auto& local = LocalPerTagAllocCounters[tag][sizeIdx];
+ auto& global = GlobalPerTagAllocCounters[tag][sizeIdx];
+ local.Flush(global);
+ }
+ }
+#endif
+#ifdef _win_
+ if (hThread)
+ CloseHandle(hThread);
+#endif
+ }
+};
+PERTHREAD TThreadAllocInfo* pThreadInfo;
+static TThreadAllocInfo* pThreadInfoList;
+
+static int* volatile nLockThreadInfo = nullptr;
+class TLockThreadListMMgr {
+public:
+ TLockThreadListMMgr() {
+ RealEnterCritical(&nLockThreadInfo);
+ }
+ ~TLockThreadListMMgr() {
+ RealLeaveCritical(&nLockThreadInfo);
+ }
+};
+
+static Y_FORCE_INLINE void IncrementCounter(ELFAllocCounter counter, size_t value) {
+#ifdef LFALLOC_YT
+ TThreadAllocInfo* thr = pThreadInfo;
+ if (thr) {
+ thr->LocalCounters[counter].Increment(value);
+ } else {
+ AtomicAdd(GlobalCounters[counter], value);
+ }
+#endif
+}
+
+extern "C" i64 GetLFAllocCounterFast(int counter) {
+#ifdef LFALLOC_YT
+ return GlobalCounters[counter];
+#else
+ return 0;
+#endif
+}
+
+extern "C" i64 GetLFAllocCounterFull(int counter) {
+#ifdef LFALLOC_YT
+ i64 ret = GlobalCounters[counter];
+ {
+ TLockThreadListMMgr ll;
+ for (TThreadAllocInfo** p = &pThreadInfoList; *p;) {
+ TThreadAllocInfo* pInfo = *p;
+ ret += pInfo->LocalCounters[counter].Value;
+ p = &pInfo->pNextInfo;
+ }
+ }
+ return ret;
+#else
+ return 0;
+#endif
+}
+
+static void MoveSingleThreadFreeToGlobal(TThreadAllocInfo* pInfo) {
+ for (int sizeIdx = 0; sizeIdx < N_SIZES; ++sizeIdx) {
+ int& freePtrIdx = pInfo->FreePtrIndex[sizeIdx];
+ char** freePtrs = pInfo->FreePtrs[sizeIdx];
+ PutBlocksToGlobalFreeList(sizeIdx, freePtrs + freePtrIdx, THREAD_BUF - freePtrIdx);
+ freePtrIdx = THREAD_BUF;
+ }
+}
+
+#ifdef _win_
+static bool IsDeadThread(TThreadAllocInfo* pInfo) {
+ DWORD dwExit;
+ bool isDead = !GetExitCodeThread(pInfo->hThread, &dwExit) || dwExit != STILL_ACTIVE;
+ return isDead;
+}
+
+static void CleanupAfterDeadThreads() {
+ TLockThreadListMMgr ls;
+ for (TThreadAllocInfo** p = &pThreadInfoList; *p;) {
+ TThreadAllocInfo* pInfo = *p;
+ if (IsDeadThread(pInfo)) {
+ MoveSingleThreadFreeToGlobal(pInfo);
+ pInfo->Done();
+ *p = pInfo->pNextInfo;
+ SystemFree(pInfo);
+ } else
+ p = &pInfo->pNextInfo;
+ }
+}
+#endif
+
+#ifndef _win_
+static pthread_key_t ThreadCacheCleaner;
+static void* volatile ThreadCacheCleanerStarted; // 0 = not started, -1 = started, -2 = is starting
+static PERTHREAD bool IsStoppingThread;
+
+static void FreeThreadCache(void*) {
+ TThreadAllocInfo* pToDelete = nullptr;
+ {
+ TLockThreadListMMgr ls;
+ pToDelete = pThreadInfo;
+ if (pToDelete == nullptr)
+ return;
+
+ // remove from the list
+ for (TThreadAllocInfo** p = &pThreadInfoList; *p; p = &(*p)->pNextInfo) {
+ if (*p == pToDelete) {
+ *p = pToDelete->pNextInfo;
+ break;
+ }
+ }
+ IsStoppingThread = true;
+ pThreadInfo = nullptr;
+ }
+
+ // free per thread buf
+ MoveSingleThreadFreeToGlobal(pToDelete);
+ pToDelete->Done();
+ SystemFree(pToDelete);
+}
+#endif
+
+static void AllocThreadInfo() {
+#ifndef _win_
+ if (DoCas(&ThreadCacheCleanerStarted, (void*)-2, (void*)nullptr) == (void*)nullptr) {
+ pthread_key_create(&ThreadCacheCleaner, FreeThreadCache);
+ ThreadCacheCleanerStarted = (void*)-1;
+ }
+ if (ThreadCacheCleanerStarted != (void*)-1)
+ return; // do not use ThreadCacheCleaner until it is constructed
+
+ {
+ if (IsStoppingThread)
+ return;
+ TLockThreadListMMgr ls;
+ if (IsStoppingThread) // better safe than sorry
+ return;
+
+ pThreadInfo = (TThreadAllocInfo*)SystemAlloc(sizeof(TThreadAllocInfo));
+ pThreadInfo->Init(&pThreadInfoList);
+ }
+ pthread_setspecific(ThreadCacheCleaner, (void*)-1); // without value destructor will not be called
+#else
+ CleanupAfterDeadThreads();
+ {
+ TLockThreadListMMgr ls;
+ pThreadInfo = (TThreadAllocInfo*)SystemAlloc(sizeof(TThreadAllocInfo));
+ pThreadInfo->Init(&pThreadInfoList);
+ }
+#endif
+}
+
+ //////////////////////////////////////////////////////////////////////////
+ // DBG stuff
+ //////////////////////////////////////////////////////////////////////////
+
+#if defined(LFALLOC_DBG)
+
+struct TAllocHeader {
+ size_t Size;
+ int Tag;
+ int Cookie;
+};
+
+static inline void* GetAllocPtr(TAllocHeader* p) {
+ return p + 1;
+}
+
+static inline TAllocHeader* GetAllocHeader(void* p) {
+ return ((TAllocHeader*)p) - 1;
+}
+
+PERTHREAD int AllocationTag;
+extern "C" int SetThreadAllocTag(int tag) {
+ int prevTag = AllocationTag;
+ if (tag < DBG_ALLOC_MAX_TAG && tag >= 0) {
+ AllocationTag = tag;
+ }
+ return prevTag;
+}
+
+PERTHREAD bool ProfileCurrentThread;
+extern "C" bool SetProfileCurrentThread(bool newVal) {
+ bool prevVal = ProfileCurrentThread;
+ ProfileCurrentThread = newVal;
+ return prevVal;
+}
+
+static volatile bool ProfileAllThreads;
+extern "C" bool SetProfileAllThreads(bool newVal) {
+ bool prevVal = ProfileAllThreads;
+ ProfileAllThreads = newVal;
+ return prevVal;
+}
+
+static volatile bool AllocationSamplingEnabled;
+extern "C" bool SetAllocationSamplingEnabled(bool newVal) {
+ bool prevVal = AllocationSamplingEnabled;
+ AllocationSamplingEnabled = newVal;
+ return prevVal;
+}
+
+static size_t AllocationSampleRate = 1000;
+extern "C" size_t SetAllocationSampleRate(size_t newVal) {
+ size_t prevVal = AllocationSampleRate;
+ AllocationSampleRate = newVal;
+ return prevVal;
+}
+
+static size_t AllocationSampleMaxSize = N_MAX_FAST_SIZE;
+extern "C" size_t SetAllocationSampleMaxSize(size_t newVal) {
+ size_t prevVal = AllocationSampleMaxSize;
+ AllocationSampleMaxSize = newVal;
+ return prevVal;
+}
+
+using TAllocationCallback = int(int tag, size_t size, int sizeIdx);
+static TAllocationCallback* AllocationCallback;
+extern "C" TAllocationCallback* SetAllocationCallback(TAllocationCallback* newVal) {
+ TAllocationCallback* prevVal = AllocationCallback;
+ AllocationCallback = newVal;
+ return prevVal;
+}
+
+using TDeallocationCallback = void(int cookie, int tag, size_t size, int sizeIdx);
+static TDeallocationCallback* DeallocationCallback;
+extern "C" TDeallocationCallback* SetDeallocationCallback(TDeallocationCallback* newVal) {
+ TDeallocationCallback* prevVal = DeallocationCallback;
+ DeallocationCallback = newVal;
+ return prevVal;
+}
+
+PERTHREAD TAtomic AllocationsCount;
+PERTHREAD bool InAllocationCallback;
+
+static const int DBG_ALLOC_INVALID_COOKIE = -1;
+static inline int SampleAllocation(TAllocHeader* p, int sizeIdx) {
+ int cookie = DBG_ALLOC_INVALID_COOKIE;
+ if (AllocationSamplingEnabled && (ProfileCurrentThread || ProfileAllThreads) && !InAllocationCallback) {
+ if (p->Size > AllocationSampleMaxSize || ++AllocationsCount % AllocationSampleRate == 0) {
+ if (AllocationCallback) {
+ InAllocationCallback = true;
+ cookie = AllocationCallback(p->Tag, p->Size, sizeIdx);
+ InAllocationCallback = false;
+ }
+ }
+ }
+ return cookie;
+}
+
+static inline void SampleDeallocation(TAllocHeader* p, int sizeIdx) {
+ if (p->Cookie != DBG_ALLOC_INVALID_COOKIE && !InAllocationCallback) {
+ if (DeallocationCallback) {
+ InAllocationCallback = true;
+ DeallocationCallback(p->Cookie, p->Tag, p->Size, sizeIdx);
+ InAllocationCallback = false;
+ }
+ }
+}
+
+static inline void TrackPerTagAllocation(TAllocHeader* p, int sizeIdx) {
+ if (p->Tag < DBG_ALLOC_MAX_TAG && p->Tag >= 0) {
+ Y_ASSERT_NOBT(sizeIdx < DBG_ALLOC_NUM_SIZES);
+ auto& global = GlobalPerTagAllocCounters[p->Tag][sizeIdx];
+
+ TThreadAllocInfo* thr = pThreadInfo;
+ if (thr) {
+ auto& local = thr->LocalPerTagAllocCounters[p->Tag][sizeIdx];
+ local.Alloc(global, p->Size);
+ } else {
+ global.Alloc(p->Size);
+ }
+ }
+}
+
+static inline void TrackPerTagDeallocation(TAllocHeader* p, int sizeIdx) {
+ if (p->Tag < DBG_ALLOC_MAX_TAG && p->Tag >= 0) {
+ Y_ASSERT_NOBT(sizeIdx < DBG_ALLOC_NUM_SIZES);
+ auto& global = GlobalPerTagAllocCounters[p->Tag][sizeIdx];
+
+ TThreadAllocInfo* thr = pThreadInfo;
+ if (thr) {
+ auto& local = thr->LocalPerTagAllocCounters[p->Tag][sizeIdx];
+ local.Free(global, p->Size);
+ } else {
+ global.Free(p->Size);
+ }
+ }
+}
+
+static void* TrackAllocation(void* ptr, size_t size, int sizeIdx) {
+ TAllocHeader* p = (TAllocHeader*)ptr;
+ p->Size = size;
+ p->Tag = AllocationTag;
+ p->Cookie = SampleAllocation(p, sizeIdx);
+ TrackPerTagAllocation(p, sizeIdx);
+ return GetAllocPtr(p);
+}
+
+static void TrackDeallocation(void* ptr, int sizeIdx) {
+ TAllocHeader* p = (TAllocHeader*)ptr;
+ SampleDeallocation(p, sizeIdx);
+ TrackPerTagDeallocation(p, sizeIdx);
+}
+
+struct TPerTagAllocInfo {
+ ssize_t Count;
+ ssize_t Size;
+};
+
+extern "C" void GetPerTagAllocInfo(
+ bool flushPerThreadCounters,
+ TPerTagAllocInfo* info,
+ int& maxTag,
+ int& numSizes) {
+ maxTag = DBG_ALLOC_MAX_TAG;
+ numSizes = DBG_ALLOC_NUM_SIZES;
+
+ if (info) {
+ if (flushPerThreadCounters) {
+ TLockThreadListMMgr ll;
+ for (TThreadAllocInfo** p = &pThreadInfoList; *p;) {
+ TThreadAllocInfo* pInfo = *p;
+ for (int tag = 0; tag < DBG_ALLOC_MAX_TAG; ++tag) {
+ for (int sizeIdx = 0; sizeIdx < DBG_ALLOC_NUM_SIZES; ++sizeIdx) {
+ auto& local = pInfo->LocalPerTagAllocCounters[tag][sizeIdx];
+ auto& global = GlobalPerTagAllocCounters[tag][sizeIdx];
+ local.Flush(global);
+ }
+ }
+ p = &pInfo->pNextInfo;
+ }
+ }
+
+ for (int tag = 0; tag < DBG_ALLOC_MAX_TAG; ++tag) {
+ for (int sizeIdx = 0; sizeIdx < DBG_ALLOC_NUM_SIZES; ++sizeIdx) {
+ auto& global = GlobalPerTagAllocCounters[tag][sizeIdx];
+ auto& res = info[tag * DBG_ALLOC_NUM_SIZES + sizeIdx];
+ res.Count = global.Count;
+ res.Size = global.Size;
+ }
+ }
+ }
+}
+
+#endif // LFALLOC_DBG
+
+//////////////////////////////////////////////////////////////////////////
+static Y_FORCE_INLINE void* LFAllocImpl(size_t _nSize) {
+#if defined(LFALLOC_DBG)
+ size_t size = _nSize;
+ _nSize += sizeof(TAllocHeader);
+#endif
+
+ IncrementCounter(CT_USER_ALLOC, _nSize);
+
+ int nSizeIdx;
+ if (_nSize > 512) {
+ if (_nSize > N_MAX_FAST_SIZE) {
+ void* ptr = LargeBlockAlloc(_nSize, CT_LARGE_ALLOC);
+#if defined(LFALLOC_DBG)
+ ptr = TrackAllocation(ptr, size, N_SIZES);
+#endif
+ return ptr;
+ }
+ nSizeIdx = size2idxArr2[(_nSize - 1) >> 8];
+ } else
+ nSizeIdx = size2idxArr1[1 + (((int)_nSize - 1) >> 3)];
+
+ IncrementCounter(CT_SMALL_ALLOC, nSizeIdxToSize[nSizeIdx]);
+
+ // check per thread buffer
+ TThreadAllocInfo* thr = pThreadInfo;
+ if (!thr) {
+ AllocThreadInfo();
+ thr = pThreadInfo;
+ if (!thr) {
+ void* ptr = LFAllocNoCache(nSizeIdx, MEM_DEFRAG);
+#if defined(LFALLOC_DBG)
+ ptr = TrackAllocation(ptr, size, nSizeIdx);
+#endif
+ return ptr;
+ }
+ }
+ {
+ int& freePtrIdx = thr->FreePtrIndex[nSizeIdx];
+ if (freePtrIdx < THREAD_BUF) {
+ void* ptr = thr->FreePtrs[nSizeIdx][freePtrIdx++];
+#if defined(LFALLOC_DBG)
+ ptr = TrackAllocation(ptr, size, nSizeIdx);
+#endif
+ return ptr;
+ }
+
+ // try to alloc from global free list
+ char* buf[FL_GROUP_SIZE];
+ int count = TakeBlocksFromGlobalFreeList(nSizeIdx, buf);
+ if (count == 0) {
+ count = LFAllocNoCacheMultiple(nSizeIdx, buf);
+ if (count == 0) {
+ NMalloc::AbortFromCorruptedAllocator(); // no way LFAllocNoCacheMultiple() can fail
+ }
+ }
+ char** dstBuf = thr->FreePtrs[nSizeIdx] + freePtrIdx - 1;
+ for (int i = 0; i < count - 1; ++i)
+ dstBuf[-i] = buf[i];
+ freePtrIdx -= count - 1;
+ void* ptr = buf[count - 1];
+#if defined(LFALLOC_DBG)
+ ptr = TrackAllocation(ptr, size, nSizeIdx);
+#endif
+ return ptr;
+ }
+}
+
+static Y_FORCE_INLINE void* LFAlloc(size_t _nSize) {
+ void* res = LFAllocImpl(_nSize);
+#ifdef DBG_FILL_MEMORY
+ if (FillMemoryOnAllocation && res && (_nSize <= DBG_FILL_MAX_SIZE)) {
+ memset(res, 0xcf, _nSize);
+ }
+#endif
+ return res;
+}
+
+static Y_FORCE_INLINE void LFFree(void* p) {
+#if defined(LFALLOC_DBG)
+ if (p == nullptr)
+ return;
+ p = GetAllocHeader(p);
+#endif
+
+ uintptr_t chkOffset = ((char*)p - ALLOC_START) - 1ll;
+ if (chkOffset >= N_MAX_WORKSET_SIZE) {
+ if (p == nullptr)
+ return;
+#if defined(LFALLOC_DBG)
+ TrackDeallocation(p, N_SIZES);
+#endif
+ LargeBlockFree(p, CT_LARGE_FREE);
+ return;
+ }
+
+ uintptr_t chunk = ((char*)p - ALLOC_START) / N_CHUNK_SIZE;
+ ptrdiff_t nSizeIdx = chunkSizeIdx[chunk];
+ if (nSizeIdx <= 0) {
+#if defined(LFALLOC_DBG)
+ TrackDeallocation(p, N_SIZES);
+#endif
+ LargeBlockFree(p, CT_LARGE_FREE);
+ return;
+ }
+
+#if defined(LFALLOC_DBG)
+ TrackDeallocation(p, nSizeIdx);
+#endif
+
+#ifdef DBG_FILL_MEMORY
+ memset(p, 0xfe, nSizeIdxToSize[nSizeIdx]);
+#endif
+
+ IncrementCounter(CT_SMALL_FREE, nSizeIdxToSize[nSizeIdx]);
+
+ // try to store info to per thread buf
+ TThreadAllocInfo* thr = pThreadInfo;
+ if (thr) {
+ int& freePtrIdx = thr->FreePtrIndex[nSizeIdx];
+ if (freePtrIdx > borderSizes[nSizeIdx]) {
+ thr->FreePtrs[nSizeIdx][--freePtrIdx] = (char*)p;
+ return;
+ }
+
+ // move several pointers to global free list
+ int freeCount = FL_GROUP_SIZE;
+ if (freeCount > THREAD_BUF - freePtrIdx)
+ freeCount = THREAD_BUF - freePtrIdx;
+ char** freePtrs = thr->FreePtrs[nSizeIdx];
+ PutBlocksToGlobalFreeList(nSizeIdx, freePtrs + freePtrIdx, freeCount);
+ freePtrIdx += freeCount;
+
+ freePtrs[--freePtrIdx] = (char*)p;
+
+ } else {
+ AllocThreadInfo();
+ PutBlocksToGlobalFreeList(nSizeIdx, (char**)&p, 1);
+ }
+}
+
+static size_t LFGetSize(const void* p) {
+#if defined(LFALLOC_DBG)
+ if (p == nullptr)
+ return 0;
+ return GetAllocHeader(const_cast(p))->Size;
+#endif
+
+ uintptr_t chkOffset = ((const char*)p - ALLOC_START);
+ if (chkOffset >= N_MAX_WORKSET_SIZE) {
+ if (p == nullptr)
+ return 0;
+ return TLargeBlk::As(p)->Pages * 4096ll;
+ }
+ uintptr_t chunk = ((const char*)p - ALLOC_START) / N_CHUNK_SIZE;
+ ptrdiff_t nSizeIdx = chunkSizeIdx[chunk];
+ if (nSizeIdx <= 0)
+ return TLargeBlk::As(p)->Pages * 4096ll;
+ return nSizeIdxToSize[nSizeIdx];
+}
+
+////////////////////////////////////////////////////////////////////////////////////////////////////
+// Output mem alloc stats
+const int N_PAGE_SIZE = 4096;
+static void DebugTraceMMgr(const char* pszFormat, ...) // __cdecl
+{
+ static char buff[20000];
+ va_list va;
+ //
+ va_start(va, pszFormat);
+ vsprintf(buff, pszFormat, va);
+ va_end(va);
+//
+#ifdef _win_
+ OutputDebugStringA(buff);
+#else
+ fprintf(stderr, buff);
+#endif
+}
+
+struct TChunkStats {
+ char *Start, *Finish;
+ i64 Size;
+ char* Entries;
+ i64 FreeCount;
+
+ TChunkStats(size_t chunk, i64 size, char* entries)
+ : Size(size)
+ , Entries(entries)
+ , FreeCount(0)
+ {
+ Start = ALLOC_START + chunk * N_CHUNK_SIZE;
+ Finish = Start + N_CHUNK_SIZE;
+ }
+ void CheckBlock(char* pBlock) {
+ if (pBlock && pBlock >= Start && pBlock < Finish) {
+ ++FreeCount;
+ i64 nShift = pBlock - Start;
+ i64 nOffsetInStep = nShift & (N_CHUNK_SIZE - 1);
+ Entries[nOffsetInStep / Size] = 1;
+ }
+ }
+ void SetGlobalFree(char* ptr) {
+ i64 nShift = ptr - Start;
+ i64 nOffsetInStep = nShift & (N_CHUNK_SIZE - 1);
+ while (nOffsetInStep + Size <= N_CHUNK_SIZE) {
+ ++FreeCount;
+ Entries[nOffsetInStep / Size] = 1;
+ nOffsetInStep += Size;
+ }
+ }
+};
+
+static void DumpMemoryBlockUtilizationLocked() {
+ TFreeListGroup* wholeLists[N_SIZES];
+ for (int nSizeIdx = 0; nSizeIdx < N_SIZES; ++nSizeIdx) {
+ wholeLists[nSizeIdx] = (TFreeListGroup*)globalFreeLists[nSizeIdx].GetWholeList();
+ }
+ char* bfList = (char*)blockFreeList.GetWholeList();
+
+ DebugTraceMMgr("memory blocks utilisation stats:\n");
+ i64 nTotalAllocated = 0, nTotalFree = 0, nTotalBadPages = 0, nTotalPages = 0, nTotalUsed = 0, nTotalLocked = 0;
+ i64 nTotalGroupBlocks = 0;
+ char* entries;
+ entries = (char*)SystemAlloc((N_CHUNK_SIZE / 4));
+ for (size_t k = 0; k < N_CHUNKS; ++k) {
+ if (chunkSizeIdx[k] <= 0) {
+ if (chunkSizeIdx[k] == -1)
+ nTotalLocked += N_CHUNK_SIZE;
+ continue;
+ }
+ i64 nSizeIdx = chunkSizeIdx[k];
+ i64 nSize = nSizeIdxToSize[nSizeIdx];
+ TChunkStats cs(k, nSize, entries);
+ int nEntriesTotal = N_CHUNK_SIZE / nSize;
+ memset(entries, 0, nEntriesTotal);
+ for (TFreeListGroup* g = wholeLists[nSizeIdx]; g; g = g->Next) {
+ for (auto& ptr : g->Ptrs)
+ cs.CheckBlock(ptr);
+ }
+ TChunkStats csGB(k, nSize, entries);
+ if (nSizeIdx == FREE_LIST_GROUP_SIZEIDX) {
+ for (auto g : wholeLists) {
+ for (; g; g = g->Next)
+ csGB.CheckBlock((char*)g);
+ }
+ for (char* blk = bfList; blk; blk = *(char**)blk)
+ csGB.CheckBlock(blk);
+ nTotalGroupBlocks += csGB.FreeCount * nSize;
+ }
+ if (((globalCurrentPtr[nSizeIdx] - ALLOC_START) / N_CHUNK_SIZE) == k)
+ cs.SetGlobalFree(globalCurrentPtr[nSizeIdx]);
+ nTotalUsed += (nEntriesTotal - cs.FreeCount - csGB.FreeCount) * nSize;
+
+ char pages[N_CHUNK_SIZE / N_PAGE_SIZE];
+ memset(pages, 0, sizeof(pages));
+ for (int i = 0, nShift = 0; i < nEntriesTotal; ++i, nShift += nSize) {
+ int nBit = 0;
+ if (entries[i])
+ nBit = 1; // free entry
+ else
+ nBit = 2; // used entry
+ for (i64 nDelta = nSize - 1; nDelta >= 0; nDelta -= N_PAGE_SIZE)
+ pages[(nShift + nDelta) / N_PAGE_SIZE] |= nBit;
+ }
+ i64 nBadPages = 0;
+ for (auto page : pages) {
+ nBadPages += page == 3;
+ nTotalPages += page != 1;
+ }
+ DebugTraceMMgr("entry = %lld; size = %lld; free = %lld; system %lld; utilisation = %g%%, fragmentation = %g%%\n",
+ k, nSize, cs.FreeCount * nSize, csGB.FreeCount * nSize,
+ (N_CHUNK_SIZE - cs.FreeCount * nSize) * 100.0f / N_CHUNK_SIZE, 100.0f * nBadPages / Y_ARRAY_SIZE(pages));
+ nTotalAllocated += N_CHUNK_SIZE;
+ nTotalFree += cs.FreeCount * nSize;
+ nTotalBadPages += nBadPages;
+ }
+ SystemFree(entries);
+ DebugTraceMMgr("Total allocated = %llu, free = %lld, system = %lld, locked for future use %lld, utilisation = %g, fragmentation = %g\n",
+ nTotalAllocated, nTotalFree, nTotalGroupBlocks, nTotalLocked,
+ 100.0f * (nTotalAllocated - nTotalFree) / nTotalAllocated, 100.0f * nTotalBadPages / nTotalPages);
+ DebugTraceMMgr("Total %lld bytes used, %lld bytes in used pages\n", nTotalUsed, nTotalPages * N_PAGE_SIZE);
+
+ for (int nSizeIdx = 0; nSizeIdx < N_SIZES; ++nSizeIdx)
+ globalFreeLists[nSizeIdx].ReturnWholeList(wholeLists[nSizeIdx]);
+ blockFreeList.ReturnWholeList(bfList);
+}
+
+void FlushThreadFreeList() {
+ if (pThreadInfo)
+ MoveSingleThreadFreeToGlobal(pThreadInfo);
+}
+
+void DumpMemoryBlockUtilization() {
+ // move current thread free to global lists to get better statistics
+ FlushThreadFreeList();
+ {
+ CCriticalSectionLockMMgr ls;
+ DumpMemoryBlockUtilizationLocked();
+ }
+}
+
+//////////////////////////////////////////////////////////////////////////
+// malloc api
+
+static bool LFAlloc_SetParam(const char* param, const char* value) {
+ if (!strcmp(param, "LB_LIMIT_TOTAL_SIZE")) {
+ LB_LIMIT_TOTAL_SIZE = atoi(value);
+ return true;
+ }
+ if (!strcmp(param, "LB_LIMIT_TOTAL_SIZE_BYTES")) {
+ LB_LIMIT_TOTAL_SIZE = (atoi(value) + N_PAGE_SIZE - 1) / N_PAGE_SIZE;
+ return true;
+ }
+#ifdef DBG_FILL_MEMORY
+ if (!strcmp(param, "FillMemoryOnAllocation")) {
+ FillMemoryOnAllocation = !strcmp(value, "true");
+ return true;
+ }
+#endif
+ if (!strcmp(param, "BeforeLFAllocGlobalLockAcquired")) {
+ BeforeLFAllocGlobalLockAcquired = (decltype(BeforeLFAllocGlobalLockAcquired))(value);
+ return true;
+ }
+ if (!strcmp(param, "AfterLFAllocGlobalLockReleased")) {
+ AfterLFAllocGlobalLockReleased = (decltype(AfterLFAllocGlobalLockReleased))(value);
+ return true;
+ }
+ if (!strcmp(param, "EnterCritical")) {
+ assert(value);
+ RealEnterCritical = (decltype(RealEnterCritical))(value);
+ return true;
+ }
+ if (!strcmp(param, "LeaveCritical")) {
+ assert(value);
+ RealLeaveCritical = (decltype(RealLeaveCritical))(value);
+ return true;
+ }
+ if (!strcmp(param, "TransparentHugePages")) {
+ TransparentHugePages = !strcmp(value, "true");
+ return true;
+ }
+ if (!strcmp(param, "MapHugeTLB")) {
+ MapHugeTLB = !strcmp(value, "true");
+ return true;
+ }
+ if (!strcmp(param, "EnableDefrag")) {
+ EnableDefrag = !strcmp(value, "true");
+ return true;
+ }
+ return false;
+};
+
+static const char* LFAlloc_GetParam(const char* param) {
+ struct TParam {
+ const char* Name;
+ const char* Value;
+ };
+
+ static const TParam Params[] = {
+ {"GetLFAllocCounterFast", (const char*)&GetLFAllocCounterFast},
+ {"GetLFAllocCounterFull", (const char*)&GetLFAllocCounterFull},
+#if defined(LFALLOC_DBG)
+ {"SetThreadAllocTag", (const char*)&SetThreadAllocTag},
+ {"SetProfileCurrentThread", (const char*)&SetProfileCurrentThread},
+ {"SetProfileAllThreads", (const char*)&SetProfileAllThreads},
+ {"SetAllocationSamplingEnabled", (const char*)&SetAllocationSamplingEnabled},
+ {"SetAllocationSampleRate", (const char*)&SetAllocationSampleRate},
+ {"SetAllocationSampleMaxSize", (const char*)&SetAllocationSampleMaxSize},
+ {"SetAllocationCallback", (const char*)&SetAllocationCallback},
+ {"SetDeallocationCallback", (const char*)&SetDeallocationCallback},
+ {"GetPerTagAllocInfo", (const char*)&GetPerTagAllocInfo},
+#endif // LFALLOC_DBG
+ };
+
+ for (int i = 0; i < Y_ARRAY_SIZE(Params); ++i) {
+ if (strcmp(param, Params[i].Name) == 0) {
+ return Params[i].Value;
+ }
+ }
+ return nullptr;
+}
+
+static Y_FORCE_INLINE void* LFVAlloc(size_t size) {
+ const size_t pg = N_PAGE_SIZE;
+ size_t bigsize = (size + pg - 1) & (~(pg - 1));
+ void* p = LFAlloc(bigsize);
+
+ Y_ASSERT_NOBT((intptr_t)p % N_PAGE_SIZE == 0);
+ return p;
+}
+
+static Y_FORCE_INLINE int LFPosixMemalign(void** memptr, size_t alignment, size_t size) {
+ if (Y_UNLIKELY(alignment > 4096)) {
+#ifdef _win_
+ OutputDebugStringA("Larger alignment are not guaranteed with this implementation\n");
+#else
+ fprintf(stderr, "Larger alignment are not guaranteed with this implementation\n");
+#endif
+ NMalloc::AbortFromCorruptedAllocator();
+ }
+ size_t bigsize = size;
+ if (bigsize <= alignment) {
+ bigsize = alignment;
+ } else if (bigsize < 2 * alignment) {
+ bigsize = 2 * alignment;
+ }
+ *memptr = LFAlloc(bigsize);
+ return 0;
+}
+#endif
diff --git a/contrib/lfalloc/src/lfmalloc.h b/contrib/lfalloc/src/lfmalloc.h
new file mode 100644
index 00000000000..1e6a0d55773
--- /dev/null
+++ b/contrib/lfalloc/src/lfmalloc.h
@@ -0,0 +1,23 @@
+#pragma once
+
+#include
+#include
+#include "util/system/compiler.h"
+
+namespace NMalloc {
+ volatile inline bool IsAllocatorCorrupted = false;
+
+ static inline void AbortFromCorruptedAllocator() {
+ IsAllocatorCorrupted = true;
+ abort();
+ }
+
+ struct TAllocHeader {
+ void* Block;
+ size_t AllocSize;
+ void Y_FORCE_INLINE Encode(void* block, size_t size, size_t signature) {
+ Block = block;
+ AllocSize = size | signature;
+ }
+ };
+}
diff --git a/contrib/lfalloc/src/util/README.md b/contrib/lfalloc/src/util/README.md
new file mode 100644
index 00000000000..c367cb4b439
--- /dev/null
+++ b/contrib/lfalloc/src/util/README.md
@@ -0,0 +1,33 @@
+Style guide for the util folder is a stricter version of general style guide (mostly in terms of ambiguity resolution).
+
+ * all {} must be in K&R style
+ * &, * tied closer to a type, not to variable
+ * always use `using` not `typedef`
+ * even a single line block must be in braces {}:
+ ```
+ if (A) {
+ B();
+ }
+ ```
+ * _ at the end of private data member of a class - `First_`, `Second_`
+ * every .h file must be accompanied with corresponding .cpp to avoid a leakage and check that it is self contained
+ * prohibited to use `printf`-like functions
+
+
+Things declared in the general style guide, which sometimes are missed:
+
+ * `template <`, not `template<`
+ * `noexcept`, not `throw ()` nor `throw()`, not required for destructors
+ * indents inside `namespace` same as inside `class`
+
+
+Requirements for a new code (and for corrections in an old code which involves change of behaviour) in util:
+
+ * presence of UNIT-tests
+ * presence of comments in Doxygen style
+ * accessors without Get prefix (`Length()`, but not `GetLength()`)
+
+This guide is not a mandatory as there is the general style guide.
+Nevertheless if it is not followed, then a next `ya style .` run in the util folder will undeservedly update authors of some lines of code.
+
+Thus before a commit it is recommended to run `ya style .` in the util folder.
diff --git a/contrib/lfalloc/src/util/system/atomic.h b/contrib/lfalloc/src/util/system/atomic.h
new file mode 100644
index 00000000000..9876515a54d
--- /dev/null
+++ b/contrib/lfalloc/src/util/system/atomic.h
@@ -0,0 +1,51 @@
+#pragma once
+
+#include "defaults.h"
+
+using TAtomicBase = intptr_t;
+using TAtomic = volatile TAtomicBase;
+
+#if defined(__GNUC__)
+#include "atomic_gcc.h"
+#elif defined(_MSC_VER)
+#include "atomic_win.h"
+#else
+#error unsupported platform
+#endif
+
+#if !defined(ATOMIC_COMPILER_BARRIER)
+#define ATOMIC_COMPILER_BARRIER()
+#endif
+
+static inline TAtomicBase AtomicSub(TAtomic& a, TAtomicBase v) {
+ return AtomicAdd(a, -v);
+}
+
+static inline TAtomicBase AtomicGetAndSub(TAtomic& a, TAtomicBase v) {
+ return AtomicGetAndAdd(a, -v);
+}
+
+#if defined(USE_GENERIC_SETGET)
+static inline TAtomicBase AtomicGet(const TAtomic& a) {
+ return a;
+}
+
+static inline void AtomicSet(TAtomic& a, TAtomicBase v) {
+ a = v;
+}
+#endif
+
+static inline bool AtomicTryLock(TAtomic* a) {
+ return AtomicCas(a, 1, 0);
+}
+
+static inline bool AtomicTryAndTryLock(TAtomic* a) {
+ return (AtomicGet(*a) == 0) && AtomicTryLock(a);
+}
+
+static inline void AtomicUnlock(TAtomic* a) {
+ ATOMIC_COMPILER_BARRIER();
+ AtomicSet(*a, 0);
+}
+
+#include "atomic_ops.h"
diff --git a/contrib/lfalloc/src/util/system/atomic_gcc.h b/contrib/lfalloc/src/util/system/atomic_gcc.h
new file mode 100644
index 00000000000..ed8dc2bdc53
--- /dev/null
+++ b/contrib/lfalloc/src/util/system/atomic_gcc.h
@@ -0,0 +1,90 @@
+#pragma once
+
+#define ATOMIC_COMPILER_BARRIER() __asm__ __volatile__("" \
+ : \
+ : \
+ : "memory")
+
+static inline TAtomicBase AtomicGet(const TAtomic& a) {
+ TAtomicBase tmp;
+#if defined(_arm64_)
+ __asm__ __volatile__(
+ "ldar %x[value], %[ptr] \n\t"
+ : [value] "=r"(tmp)
+ : [ptr] "Q"(a)
+ : "memory");
+#else
+ __atomic_load(&a, &tmp, __ATOMIC_ACQUIRE);
+#endif
+ return tmp;
+}
+
+static inline void AtomicSet(TAtomic& a, TAtomicBase v) {
+#if defined(_arm64_)
+ __asm__ __volatile__(
+ "stlr %x[value], %[ptr] \n\t"
+ : [ptr] "=Q"(a)
+ : [value] "r"(v)
+ : "memory");
+#else
+ __atomic_store(&a, &v, __ATOMIC_RELEASE);
+#endif
+}
+
+static inline intptr_t AtomicIncrement(TAtomic& p) {
+ return __atomic_add_fetch(&p, 1, __ATOMIC_SEQ_CST);
+}
+
+static inline intptr_t AtomicGetAndIncrement(TAtomic& p) {
+ return __atomic_fetch_add(&p, 1, __ATOMIC_SEQ_CST);
+}
+
+static inline intptr_t AtomicDecrement(TAtomic& p) {
+ return __atomic_sub_fetch(&p, 1, __ATOMIC_SEQ_CST);
+}
+
+static inline intptr_t AtomicGetAndDecrement(TAtomic& p) {
+ return __atomic_fetch_sub(&p, 1, __ATOMIC_SEQ_CST);
+}
+
+static inline intptr_t AtomicAdd(TAtomic& p, intptr_t v) {
+ return __atomic_add_fetch(&p, v, __ATOMIC_SEQ_CST);
+}
+
+static inline intptr_t AtomicGetAndAdd(TAtomic& p, intptr_t v) {
+ return __atomic_fetch_add(&p, v, __ATOMIC_SEQ_CST);
+}
+
+static inline intptr_t AtomicSwap(TAtomic* p, intptr_t v) {
+ (void)p; // disable strange 'parameter set but not used' warning on gcc
+ intptr_t ret;
+ __atomic_exchange(p, &v, &ret, __ATOMIC_SEQ_CST);
+ return ret;
+}
+
+static inline bool AtomicCas(TAtomic* a, intptr_t exchange, intptr_t compare) {
+ (void)a; // disable strange 'parameter set but not used' warning on gcc
+ return __atomic_compare_exchange(a, &compare, &exchange, false, __ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST);
+}
+
+static inline intptr_t AtomicGetAndCas(TAtomic* a, intptr_t exchange, intptr_t compare) {
+ (void)a; // disable strange 'parameter set but not used' warning on gcc
+ __atomic_compare_exchange(a, &compare, &exchange, false, __ATOMIC_SEQ_CST, __ATOMIC_SEQ_CST);
+ return compare;
+}
+
+static inline intptr_t AtomicOr(TAtomic& a, intptr_t b) {
+ return __atomic_or_fetch(&a, b, __ATOMIC_SEQ_CST);
+}
+
+static inline intptr_t AtomicXor(TAtomic& a, intptr_t b) {
+ return __atomic_xor_fetch(&a, b, __ATOMIC_SEQ_CST);
+}
+
+static inline intptr_t AtomicAnd(TAtomic& a, intptr_t b) {
+ return __atomic_and_fetch(&a, b, __ATOMIC_SEQ_CST);
+}
+
+static inline void AtomicBarrier() {
+ __sync_synchronize();
+}
diff --git a/contrib/lfalloc/src/util/system/atomic_ops.h b/contrib/lfalloc/src/util/system/atomic_ops.h
new file mode 100644
index 00000000000..425b643e14d
--- /dev/null
+++ b/contrib/lfalloc/src/util/system/atomic_ops.h
@@ -0,0 +1,189 @@
+#pragma once
+
+#include
+
+template
+inline TAtomic* AsAtomicPtr(T volatile* target) {
+ return reinterpret_cast(target);
+}
+
+template
+inline const TAtomic* AsAtomicPtr(T const volatile* target) {
+ return reinterpret_cast(target);
+}
+
+// integral types
+
+template
+struct TAtomicTraits {
+ enum {
+ Castable = std::is_integral::value && sizeof(T) == sizeof(TAtomicBase) && !std::is_const::value,
+ };
+};
+
+template
+using TEnableIfCastable = std::enable_if_t::Castable, TT>;
+
+template
+inline TEnableIfCastable AtomicGet(T const volatile& target) {
+ return static_cast(AtomicGet(*AsAtomicPtr(&target)));
+}
+
+template
+inline TEnableIfCastable AtomicSet(T volatile& target, TAtomicBase value) {
+ AtomicSet(*AsAtomicPtr(&target), value);
+}
+
+template
+inline TEnableIfCastable AtomicIncrement(T volatile& target) {
+ return static_cast(AtomicIncrement(*AsAtomicPtr(&target)));
+}
+
+template
+inline TEnableIfCastable AtomicGetAndIncrement(T volatile& target) {
+ return static_cast(AtomicGetAndIncrement(*AsAtomicPtr(&target)));
+}
+
+template
+inline TEnableIfCastable AtomicDecrement(T volatile& target) {
+ return static_cast(AtomicDecrement(*AsAtomicPtr(&target)));
+}
+
+template
+inline TEnableIfCastable AtomicGetAndDecrement(T volatile& target) {
+ return static_cast(AtomicGetAndDecrement(*AsAtomicPtr(&target)));
+}
+
+template
+inline TEnableIfCastable AtomicAdd(T volatile& target, TAtomicBase value) {
+ return static_cast(AtomicAdd(*AsAtomicPtr(&target), value));
+}
+
+template
+inline TEnableIfCastable AtomicGetAndAdd(T volatile& target, TAtomicBase value) {
+ return static_cast(AtomicGetAndAdd(*AsAtomicPtr(&target), value));
+}
+
+template
+inline TEnableIfCastable AtomicSub(T volatile& target, TAtomicBase value) {
+ return static_cast(AtomicSub(*AsAtomicPtr(&target), value));
+}
+
+template
+inline TEnableIfCastable AtomicGetAndSub(T volatile& target, TAtomicBase value) {
+ return static_cast(AtomicGetAndSub(*AsAtomicPtr(&target), value));
+}
+
+template
+inline TEnableIfCastable AtomicSwap(T volatile* target, TAtomicBase exchange) {
+ return static_cast(AtomicSwap(AsAtomicPtr(target), exchange));
+}
+
+template
+inline TEnableIfCastable AtomicCas(T volatile* target, TAtomicBase exchange, TAtomicBase compare) {
+ return AtomicCas(AsAtomicPtr(target), exchange, compare);
+}
+
+template
+inline TEnableIfCastable AtomicGetAndCas(T volatile* target, TAtomicBase exchange, TAtomicBase compare) {
+ return static_cast(AtomicGetAndCas(AsAtomicPtr(target), exchange, compare));
+}
+
+template
+inline TEnableIfCastable AtomicTryLock(T volatile* target) {
+ return AtomicTryLock(AsAtomicPtr(target));
+}
+
+template
+inline TEnableIfCastable AtomicTryAndTryLock(T volatile* target) {
+ return AtomicTryAndTryLock(AsAtomicPtr(target));
+}
+
+template
+inline TEnableIfCastable AtomicUnlock(T volatile* target) {
+ AtomicUnlock(AsAtomicPtr(target));
+}
+
+template
+inline TEnableIfCastable AtomicOr(T volatile& target, TAtomicBase value) {
+ return static_cast(AtomicOr(*AsAtomicPtr(&target), value));
+}
+
+template
+inline TEnableIfCastable AtomicAnd(T volatile& target, TAtomicBase value) {
+ return static_cast(AtomicAnd(*AsAtomicPtr(&target), value));
+}
+
+template
+inline TEnableIfCastable AtomicXor(T volatile& target, TAtomicBase value) {
+ return static_cast(AtomicXor(*AsAtomicPtr(&target), value));
+}
+
+// pointer types
+
+template
+inline T* AtomicGet(T* const volatile& target) {
+ return reinterpret_cast(AtomicGet(*AsAtomicPtr(&target)));
+}
+
+template
+inline void AtomicSet(T* volatile& target, T* value) {
+ AtomicSet(*AsAtomicPtr(&target), reinterpret_cast(value));
+}
+
+using TNullPtr = decltype(nullptr);
+
+template
+inline void AtomicSet(T* volatile& target, TNullPtr) {
+ AtomicSet(*AsAtomicPtr(&target), 0);
+}
+
+template
+inline T* AtomicSwap(T* volatile* target, T* exchange) {
+ return reinterpret_cast(AtomicSwap(AsAtomicPtr(target), reinterpret_cast(exchange)));
+}
+
+template
+inline T* AtomicSwap(T* volatile* target, TNullPtr) {
+ return reinterpret_cast(AtomicSwap(AsAtomicPtr(target), 0));
+}
+
+template
+inline bool AtomicCas(T* volatile* target, T* exchange, T* compare) {
+ return AtomicCas(AsAtomicPtr(target), reinterpret_cast(exchange), reinterpret_cast(compare));
+}
+
+template
+inline T* AtomicGetAndCas(T* volatile* target, T* exchange, T* compare) {
+ return reinterpret_cast(AtomicGetAndCas(AsAtomicPtr(target), reinterpret_cast(exchange), reinterpret_cast(compare)));
+}
+
+template
+inline bool AtomicCas(T* volatile* target, T* exchange, TNullPtr) {
+ return AtomicCas(AsAtomicPtr(target), reinterpret_cast(exchange), 0);
+}
+
+template
+inline T* AtomicGetAndCas(T* volatile* target, T* exchange, TNullPtr) {
+ return reinterpret_cast(AtomicGetAndCas(AsAtomicPtr(target), reinterpret_cast(exchange), 0));
+}
+
+template
+inline bool AtomicCas(T* volatile* target, TNullPtr, T* compare) {
+ return AtomicCas(AsAtomicPtr(target), 0, reinterpret_cast(compare));
+}
+
+template
+inline T* AtomicGetAndCas(T* volatile* target, TNullPtr, T* compare) {
+ return reinterpret_cast(AtomicGetAndCas(AsAtomicPtr(target), 0, reinterpret_cast(compare)));
+}
+
+template
+inline bool AtomicCas(T* volatile* target, TNullPtr, TNullPtr) {
+ return AtomicCas(AsAtomicPtr(target), 0, 0);
+}
+
+template
+inline T* AtomicGetAndCas(T* volatile* target, TNullPtr, TNullPtr) {
+ return reinterpret_cast(AtomicGetAndCas(AsAtomicPtr(target), 0, 0));
+}
diff --git a/contrib/lfalloc/src/util/system/atomic_win.h b/contrib/lfalloc/src/util/system/atomic_win.h
new file mode 100644
index 00000000000..1abebd87b38
--- /dev/null
+++ b/contrib/lfalloc/src/util/system/atomic_win.h
@@ -0,0 +1,114 @@
+#pragma once
+
+#include
+
+#define USE_GENERIC_SETGET
+
+#if defined(_i386_)
+
+#pragma intrinsic(_InterlockedIncrement)
+#pragma intrinsic(_InterlockedDecrement)
+#pragma intrinsic(_InterlockedExchangeAdd)
+#pragma intrinsic(_InterlockedExchange)
+#pragma intrinsic(_InterlockedCompareExchange)
+
+static inline intptr_t AtomicIncrement(TAtomic& a) {
+ return _InterlockedIncrement((volatile long*)&a);
+}
+
+static inline intptr_t AtomicGetAndIncrement(TAtomic& a) {
+ return _InterlockedIncrement((volatile long*)&a) - 1;
+}
+
+static inline intptr_t AtomicDecrement(TAtomic& a) {
+ return _InterlockedDecrement((volatile long*)&a);
+}
+
+static inline intptr_t AtomicGetAndDecrement(TAtomic& a) {
+ return _InterlockedDecrement((volatile long*)&a) + 1;
+}
+
+static inline intptr_t AtomicAdd(TAtomic& a, intptr_t b) {
+ return _InterlockedExchangeAdd((volatile long*)&a, b) + b;
+}
+
+static inline intptr_t AtomicGetAndAdd(TAtomic& a, intptr_t b) {
+ return _InterlockedExchangeAdd((volatile long*)&a, b);
+}
+
+static inline intptr_t AtomicSwap(TAtomic* a, intptr_t b) {
+ return _InterlockedExchange((volatile long*)a, b);
+}
+
+static inline bool AtomicCas(TAtomic* a, intptr_t exchange, intptr_t compare) {
+ return _InterlockedCompareExchange((volatile long*)a, exchange, compare) == compare;
+}
+
+static inline intptr_t AtomicGetAndCas(TAtomic* a, intptr_t exchange, intptr_t compare) {
+ return _InterlockedCompareExchange((volatile long*)a, exchange, compare);
+}
+
+#else // _x86_64_
+
+#pragma intrinsic(_InterlockedIncrement64)
+#pragma intrinsic(_InterlockedDecrement64)
+#pragma intrinsic(_InterlockedExchangeAdd64)
+#pragma intrinsic(_InterlockedExchange64)
+#pragma intrinsic(_InterlockedCompareExchange64)
+
+static inline intptr_t AtomicIncrement(TAtomic& a) {
+ return _InterlockedIncrement64((volatile __int64*)&a);
+}
+
+static inline intptr_t AtomicGetAndIncrement(TAtomic& a) {
+ return _InterlockedIncrement64((volatile __int64*)&a) - 1;
+}
+
+static inline intptr_t AtomicDecrement(TAtomic& a) {
+ return _InterlockedDecrement64((volatile __int64*)&a);
+}
+
+static inline intptr_t AtomicGetAndDecrement(TAtomic& a) {
+ return _InterlockedDecrement64((volatile __int64*)&a) + 1;
+}
+
+static inline intptr_t AtomicAdd(TAtomic& a, intptr_t b) {
+ return _InterlockedExchangeAdd64((volatile __int64*)&a, b) + b;
+}
+
+static inline intptr_t AtomicGetAndAdd(TAtomic& a, intptr_t b) {
+ return _InterlockedExchangeAdd64((volatile __int64*)&a, b);
+}
+
+static inline intptr_t AtomicSwap(TAtomic* a, intptr_t b) {
+ return _InterlockedExchange64((volatile __int64*)a, b);
+}
+
+static inline bool AtomicCas(TAtomic* a, intptr_t exchange, intptr_t compare) {
+ return _InterlockedCompareExchange64((volatile __int64*)a, exchange, compare) == compare;
+}
+
+static inline intptr_t AtomicGetAndCas(TAtomic* a, intptr_t exchange, intptr_t compare) {
+ return _InterlockedCompareExchange64((volatile __int64*)a, exchange, compare);
+}
+
+static inline intptr_t AtomicOr(TAtomic& a, intptr_t b) {
+ return _InterlockedOr64(&a, b) | b;
+}
+
+static inline intptr_t AtomicAnd(TAtomic& a, intptr_t b) {
+ return _InterlockedAnd64(&a, b) & b;
+}
+
+static inline intptr_t AtomicXor(TAtomic& a, intptr_t b) {
+ return _InterlockedXor64(&a, b) ^ b;
+}
+
+#endif // _x86_
+
+//TODO
+static inline void AtomicBarrier() {
+ TAtomic val = 0;
+
+ AtomicSwap(&val, 0);
+}
diff --git a/contrib/lfalloc/src/util/system/compiler.h b/contrib/lfalloc/src/util/system/compiler.h
new file mode 100644
index 00000000000..b5cec600923
--- /dev/null
+++ b/contrib/lfalloc/src/util/system/compiler.h
@@ -0,0 +1,617 @@
+#pragma once
+
+// useful cross-platfrom definitions for compilers
+
+/**
+ * @def Y_FUNC_SIGNATURE
+ *
+ * Use this macro to get pretty function name (see example).
+ *
+ * @code
+ * void Hi() {
+ * Cout << Y_FUNC_SIGNATURE << Endl;
+ * }
+
+ * template
+ * void Do() {
+ * Cout << Y_FUNC_SIGNATURE << Endl;
+ * }
+
+ * int main() {
+ * Hi(); // void Hi()
+ * Do(); // void Do() [T = int]
+ * Do(); // void Do() [T = TString]
+ * }
+ * @endcode
+ */
+#if defined(__GNUC__)
+#define Y_FUNC_SIGNATURE __PRETTY_FUNCTION__
+#elif defined(_MSC_VER)
+#define Y_FUNC_SIGNATURE __FUNCSIG__
+#else
+#define Y_FUNC_SIGNATURE ""
+#endif
+
+#ifdef __GNUC__
+#define Y_PRINTF_FORMAT(n, m) __attribute__((__format__(__printf__, n, m)))
+#endif
+
+#ifndef Y_PRINTF_FORMAT
+#define Y_PRINTF_FORMAT(n, m)
+#endif
+
+#if defined(__clang__)
+#define Y_NO_SANITIZE(...) __attribute__((no_sanitize(__VA_ARGS__)))
+#endif
+
+#if !defined(Y_NO_SANITIZE)
+#define Y_NO_SANITIZE(...)
+#endif
+
+/**
+ * @def Y_DECLARE_UNUSED
+ *
+ * Macro is needed to silence compiler warning about unused entities (e.g. function or argument).
+ *
+ * @code
+ * Y_DECLARE_UNUSED int FunctionUsedSolelyForDebugPurposes();
+ * assert(FunctionUsedSolelyForDebugPurposes() == 42);
+ *
+ * void Foo(const int argumentUsedOnlyForDebugPurposes Y_DECLARE_UNUSED) {
+ * assert(argumentUsedOnlyForDebugPurposes == 42);
+ * // however you may as well omit `Y_DECLARE_UNUSED` and use `UNUSED` macro instead
+ * Y_UNUSED(argumentUsedOnlyForDebugPurposes);
+ * }
+ * @endcode
+ */
+#ifdef __GNUC__
+#define Y_DECLARE_UNUSED __attribute__((unused))
+#endif
+
+#ifndef Y_DECLARE_UNUSED
+#define Y_DECLARE_UNUSED
+#endif
+
+#if defined(__GNUC__)
+#define Y_LIKELY(Cond) __builtin_expect(!!(Cond), 1)
+#define Y_UNLIKELY(Cond) __builtin_expect(!!(Cond), 0)
+#define Y_PREFETCH_READ(Pointer, Priority) __builtin_prefetch((const void*)(Pointer), 0, Priority)
+#define Y_PREFETCH_WRITE(Pointer, Priority) __builtin_prefetch((const void*)(Pointer), 1, Priority)
+#endif
+
+/**
+ * @def Y_FORCE_INLINE
+ *
+ * Macro to use in place of 'inline' in function declaration/definition to force
+ * it to be inlined.
+ */
+#if !defined(Y_FORCE_INLINE)
+#if defined(CLANG_COVERAGE)
+#/* excessive __always_inline__ might significantly slow down compilation of an instrumented unit */
+#define Y_FORCE_INLINE inline
+#elif defined(_MSC_VER)
+#define Y_FORCE_INLINE __forceinline
+#elif defined(__GNUC__)
+#/* Clang also defines __GNUC__ (as 4) */
+#define Y_FORCE_INLINE inline __attribute__((__always_inline__))
+#else
+#define Y_FORCE_INLINE inline
+#endif
+#endif
+
+/**
+ * @def Y_NO_INLINE
+ *
+ * Macro to use in place of 'inline' in function declaration/definition to
+ * prevent it from being inlined.
+ */
+#if !defined(Y_NO_INLINE)
+#if defined(_MSC_VER)
+#define Y_NO_INLINE __declspec(noinline)
+#elif defined(__GNUC__) || defined(__INTEL_COMPILER)
+#/* Clang also defines __GNUC__ (as 4) */
+#define Y_NO_INLINE __attribute__((__noinline__))
+#else
+#define Y_NO_INLINE
+#endif
+#endif
+
+//to cheat compiler about strict aliasing or similar problems
+#if defined(__GNUC__)
+#define Y_FAKE_READ(X) \
+ do { \
+ __asm__ __volatile__("" \
+ : \
+ : "m"(X)); \
+ } while (0)
+
+#define Y_FAKE_WRITE(X) \
+ do { \
+ __asm__ __volatile__("" \
+ : "=m"(X)); \
+ } while (0)
+#endif
+
+#if !defined(Y_FAKE_READ)
+#define Y_FAKE_READ(X)
+#endif
+
+#if !defined(Y_FAKE_WRITE)
+#define Y_FAKE_WRITE(X)
+#endif
+
+#ifndef Y_PREFETCH_READ
+#define Y_PREFETCH_READ(Pointer, Priority) (void)(const void*)(Pointer), (void)Priority
+#endif
+
+#ifndef Y_PREFETCH_WRITE
+#define Y_PREFETCH_WRITE(Pointer, Priority) (void)(const void*)(Pointer), (void)Priority
+#endif
+
+#ifndef Y_LIKELY
+#define Y_LIKELY(Cond) (Cond)
+#define Y_UNLIKELY(Cond) (Cond)
+#endif
+
+#ifdef __GNUC__
+#define _packed __attribute__((packed))
+#else
+#define _packed
+#endif
+
+#if defined(__GNUC__)
+#define Y_WARN_UNUSED_RESULT __attribute__((warn_unused_result))
+#endif
+
+#ifndef Y_WARN_UNUSED_RESULT
+#define Y_WARN_UNUSED_RESULT
+#endif
+
+#if defined(__GNUC__)
+#define Y_HIDDEN __attribute__((visibility("hidden")))
+#endif
+
+#if !defined(Y_HIDDEN)
+#define Y_HIDDEN
+#endif
+
+#if defined(__GNUC__)
+#define Y_PUBLIC __attribute__((visibility("default")))
+#endif
+
+#if !defined(Y_PUBLIC)
+#define Y_PUBLIC
+#endif
+
+#if !defined(Y_UNUSED) && !defined(__cplusplus)
+#define Y_UNUSED(var) (void)(var)
+#endif
+#if !defined(Y_UNUSED) && defined(__cplusplus)
+template
+constexpr Y_FORCE_INLINE int Y_UNUSED(Types&&...) {
+ return 0;
+};
+#endif
+
+/**
+ * @def Y_ASSUME
+ *
+ * Macro that tells the compiler that it can generate optimized code
+ * as if the given expression will always evaluate true.
+ * The behavior is undefined if it ever evaluates false.
+ *
+ * @code
+ * // factored into a function so that it's testable
+ * inline int Avg(int x, int y) {
+ * if (x >= 0 && y >= 0) {
+ * return (static_cast(x) + static_cast(y)) >> 1;
+ * } else {
+ * // a slower implementation
+ * }
+ * }
+ *
+ * // we know that xs and ys are non-negative from domain knowledge,
+ * // but we can't change the types of xs and ys because of API constrains
+ * int Foo(const TVector& xs, const TVector& ys) {
+ * TVector avgs;
+ * avgs.resize(xs.size());
+ * for (size_t i = 0; i < xs.size(); ++i) {
+ * auto x = xs[i];
+ * auto y = ys[i];
+ * Y_ASSUME(x >= 0);
+ * Y_ASSUME(y >= 0);
+ * xs[i] = Avg(x, y);
+ * }
+ * }
+ * @endcode
+ */
+#if defined(__GNUC__)
+#define Y_ASSUME(condition) ((condition) ? (void)0 : __builtin_unreachable())
+#elif defined(_MSC_VER)
+#define Y_ASSUME(condition) __assume(condition)
+#else
+#define Y_ASSUME(condition) Y_UNUSED(condition)
+#endif
+
+#ifdef __cplusplus
+[[noreturn]]
+#endif
+Y_HIDDEN void _YandexAbort();
+
+/**
+ * @def Y_UNREACHABLE
+ *
+ * Macro that marks the rest of the code branch unreachable.
+ * The behavior is undefined if it's ever reached.
+ *
+ * @code
+ * switch (i % 3) {
+ * case 0:
+ * return foo;
+ * case 1:
+ * return bar;
+ * case 2:
+ * return baz;
+ * default:
+ * Y_UNREACHABLE();
+ * }
+ * @endcode
+ */
+#if defined(__GNUC__) || defined(_MSC_VER)
+#define Y_UNREACHABLE() Y_ASSUME(0)
+#else
+#define Y_UNREACHABLE() _YandexAbort()
+#endif
+
+#if defined(undefined_sanitizer_enabled)
+#define _ubsan_enabled_
+#endif
+
+#ifdef __clang__
+
+#if __has_feature(thread_sanitizer)
+#define _tsan_enabled_
+#endif
+#if __has_feature(memory_sanitizer)
+#define _msan_enabled_
+#endif
+#if __has_feature(address_sanitizer)
+#define _asan_enabled_
+#endif
+
+#else
+
+#if defined(thread_sanitizer_enabled) || defined(__SANITIZE_THREAD__)
+#define _tsan_enabled_
+#endif
+#if defined(memory_sanitizer_enabled)
+#define _msan_enabled_
+#endif
+#if defined(address_sanitizer_enabled) || defined(__SANITIZE_ADDRESS__)
+#define _asan_enabled_
+#endif
+
+#endif
+
+#if defined(_asan_enabled_) || defined(_msan_enabled_) || defined(_tsan_enabled_) || defined(_ubsan_enabled_)
+#define _san_enabled_
+#endif
+
+#if defined(_MSC_VER)
+#define __PRETTY_FUNCTION__ __FUNCSIG__
+#endif
+
+#if defined(__GNUC__)
+#define Y_WEAK __attribute__((weak))
+#else
+#define Y_WEAK
+#endif
+
+#if defined(__CUDACC_VER_MAJOR__)
+#define Y_CUDA_AT_LEAST(x, y) (__CUDACC_VER_MAJOR__ > x || (__CUDACC_VER_MAJOR__ == x && __CUDACC_VER_MINOR__ >= y))
+#else
+#define Y_CUDA_AT_LEAST(x, y) 0
+#endif
+
+// NVidia CUDA C++ Compiler did not know about noexcept keyword until version 9.0
+#if !Y_CUDA_AT_LEAST(9, 0)
+#if defined(__CUDACC__) && !defined(noexcept)
+#define noexcept throw ()
+#endif
+#endif
+
+#if defined(__GNUC__)
+#define Y_COLD __attribute__((cold))
+#define Y_LEAF __attribute__((leaf))
+#define Y_WRAPPER __attribute__((artificial))
+#else
+#define Y_COLD
+#define Y_LEAF
+#define Y_WRAPPER
+#endif
+
+/**
+ * @def Y_PRAGMA
+ *
+ * Macro for use in other macros to define compiler pragma
+ * See below for other usage examples
+ *
+ * @code
+ * #if defined(__clang__) || defined(__GNUC__)
+ * #define Y_PRAGMA_NO_WSHADOW \
+ * Y_PRAGMA("GCC diagnostic ignored \"-Wshadow\"")
+ * #elif defined(_MSC_VER)
+ * #define Y_PRAGMA_NO_WSHADOW \
+ * Y_PRAGMA("warning(disable:4456 4457")
+ * #else
+ * #define Y_PRAGMA_NO_WSHADOW
+ * #endif
+ * @endcode
+ */
+#if defined(__clang__) || defined(__GNUC__)
+#define Y_PRAGMA(x) _Pragma(x)
+#elif defined(_MSC_VER)
+#define Y_PRAGMA(x) __pragma(x)
+#else
+#define Y_PRAGMA(x)
+#endif
+
+/**
+ * @def Y_PRAGMA_DIAGNOSTIC_PUSH
+ *
+ * Cross-compiler pragma to save diagnostic settings
+ *
+ * @see
+ * GCC: https://gcc.gnu.org/onlinedocs/gcc/Diagnostic-Pragmas.html
+ * MSVC: https://msdn.microsoft.com/en-us/library/2c8f766e.aspx
+ * Clang: https://clang.llvm.org/docs/UsersManual.html#controlling-diagnostics-via-pragmas
+ *
+ * @code
+ * Y_PRAGMA_DIAGNOSTIC_PUSH
+ * @endcode
+ */
+#if defined(__clang__) || defined(__GNUC__)
+#define Y_PRAGMA_DIAGNOSTIC_PUSH \
+ Y_PRAGMA("GCC diagnostic push")
+#elif defined(_MSC_VER)
+#define Y_PRAGMA_DIAGNOSTIC_PUSH \
+ Y_PRAGMA(warning(push))
+#else
+#define Y_PRAGMA_DIAGNOSTIC_PUSH
+#endif
+
+/**
+ * @def Y_PRAGMA_DIAGNOSTIC_POP
+ *
+ * Cross-compiler pragma to restore diagnostic settings
+ *
+ * @see
+ * GCC: https://gcc.gnu.org/onlinedocs/gcc/Diagnostic-Pragmas.html
+ * MSVC: https://msdn.microsoft.com/en-us/library/2c8f766e.aspx
+ * Clang: https://clang.llvm.org/docs/UsersManual.html#controlling-diagnostics-via-pragmas
+ *
+ * @code
+ * Y_PRAGMA_DIAGNOSTIC_POP
+ * @endcode
+ */
+#if defined(__clang__) || defined(__GNUC__)
+#define Y_PRAGMA_DIAGNOSTIC_POP \
+ Y_PRAGMA("GCC diagnostic pop")
+#elif defined(_MSC_VER)
+#define Y_PRAGMA_DIAGNOSTIC_POP \
+ Y_PRAGMA(warning(pop))
+#else
+#define Y_PRAGMA_DIAGNOSTIC_POP
+#endif
+
+/**
+ * @def Y_PRAGMA_NO_WSHADOW
+ *
+ * Cross-compiler pragma to disable warnings about shadowing variables
+ *
+ * @code
+ * Y_PRAGMA_DIAGNOSTIC_PUSH
+ * Y_PRAGMA_NO_WSHADOW
+ *
+ * // some code which use variable shadowing, e.g.:
+ *
+ * for (int i = 0; i < 100; ++i) {
+ * Use(i);
+ *
+ * for (int i = 42; i < 100500; ++i) { // this i is shadowing previous i
+ * AnotherUse(i);
+ * }
+ * }
+ *
+ * Y_PRAGMA_DIAGNOSTIC_POP
+ * @endcode
+ */
+#if defined(__clang__) || defined(__GNUC__)
+#define Y_PRAGMA_NO_WSHADOW \
+ Y_PRAGMA("GCC diagnostic ignored \"-Wshadow\"")
+#elif defined(_MSC_VER)
+#define Y_PRAGMA_NO_WSHADOW \
+ Y_PRAGMA(warning(disable : 4456 4457))
+#else
+#define Y_PRAGMA_NO_WSHADOW
+#endif
+
+/**
+ * @ def Y_PRAGMA_NO_UNUSED_FUNCTION
+ *
+ * Cross-compiler pragma to disable warnings about unused functions
+ *
+ * @see
+ * GCC: https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html
+ * Clang: https://clang.llvm.org/docs/DiagnosticsReference.html#wunused-function
+ * MSVC: there is no such warning
+ *
+ * @code
+ * Y_PRAGMA_DIAGNOSTIC_PUSH
+ * Y_PRAGMA_NO_UNUSED_FUNCTION
+ *
+ * // some code which introduces a function which later will not be used, e.g.:
+ *
+ * void Foo() {
+ * }
+ *
+ * int main() {
+ * return 0; // Foo() never called
+ * }
+ *
+ * Y_PRAGMA_DIAGNOSTIC_POP
+ * @endcode
+ */
+#if defined(__clang__) || defined(__GNUC__)
+#define Y_PRAGMA_NO_UNUSED_FUNCTION \
+ Y_PRAGMA("GCC diagnostic ignored \"-Wunused-function\"")
+#else
+#define Y_PRAGMA_NO_UNUSED_FUNCTION
+#endif
+
+/**
+ * @ def Y_PRAGMA_NO_UNUSED_PARAMETER
+ *
+ * Cross-compiler pragma to disable warnings about unused function parameters
+ *
+ * @see
+ * GCC: https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html
+ * Clang: https://clang.llvm.org/docs/DiagnosticsReference.html#wunused-parameter
+ * MSVC: https://msdn.microsoft.com/en-us/library/26kb9fy0.aspx
+ *
+ * @code
+ * Y_PRAGMA_DIAGNOSTIC_PUSH
+ * Y_PRAGMA_NO_UNUSED_PARAMETER
+ *
+ * // some code which introduces a function with unused parameter, e.g.:
+ *
+ * void foo(int a) {
+ * // a is not referenced
+ * }
+ *
+ * int main() {
+ * foo(1);
+ * return 0;
+ * }
+ *
+ * Y_PRAGMA_DIAGNOSTIC_POP
+ * @endcode
+ */
+#if defined(__clang__) || defined(__GNUC__)
+#define Y_PRAGMA_NO_UNUSED_PARAMETER \
+ Y_PRAGMA("GCC diagnostic ignored \"-Wunused-parameter\"")
+#elif defined(_MSC_VER)
+#define Y_PRAGMA_NO_UNUSED_PARAMETER \
+ Y_PRAGMA(warning(disable : 4100))
+#else
+#define Y_PRAGMA_NO_UNUSED_PARAMETER
+#endif
+
+/**
+ * @def Y_PRAGMA_NO_DEPRECATED
+ *
+ * Cross compiler pragma to disable warnings and errors about deprecated
+ *
+ * @see
+ * GCC: https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html
+ * Clang: https://clang.llvm.org/docs/DiagnosticsReference.html#wdeprecated
+ * MSVC: https://docs.microsoft.com/en-us/cpp/error-messages/compiler-warnings/compiler-warning-level-3-c4996?view=vs-2017
+ *
+ * @code
+ * Y_PRAGMA_DIAGNOSTIC_PUSH
+ * Y_PRAGMA_NO_DEPRECATED
+ *
+ * [deprecated] void foo() {
+ * // ...
+ * }
+ *
+ * int main() {
+ * foo();
+ * return 0;
+ * }
+ *
+ * Y_PRAGMA_DIAGNOSTIC_POP
+ * @endcode
+ */
+#if defined(__clang__) || defined(__GNUC__)
+#define Y_PRAGMA_NO_DEPRECATED \
+ Y_PRAGMA("GCC diagnostic ignored \"-Wdeprecated\"")
+#elif defined(_MSC_VER)
+#define Y_PRAGMA_NO_DEPRECATED \
+ Y_PRAGMA(warning(disable : 4996))
+#else
+#define Y_PRAGMA_NO_DEPRECATED
+#endif
+
+#if defined(__clang__) || defined(__GNUC__)
+/**
+ * @def Y_CONST_FUNCTION
+ methods and functions, marked with this method are promised to:
+ 1. do not have side effects
+ 2. this method do not read global memory
+ NOTE: this attribute can't be set for methods that depend on data, pointed by this
+ this allow compilers to do hard optimization of that functions
+ NOTE: in common case this attribute can't be set if method have pointer-arguments
+ NOTE: as result there no any reason to discard result of such method
+*/
+#define Y_CONST_FUNCTION [[gnu::const]]
+#endif
+
+#if !defined(Y_CONST_FUNCTION)
+#define Y_CONST_FUNCTION
+#endif
+
+#if defined(__clang__) || defined(__GNUC__)
+/**
+ * @def Y_PURE_FUNCTION
+ methods and functions, marked with this method are promised to:
+ 1. do not have side effects
+ 2. result will be the same if no global memory changed
+ this allow compilers to do hard optimization of that functions
+ NOTE: as result there no any reason to discard result of such method
+*/
+#define Y_PURE_FUNCTION [[gnu::pure]]
+#endif
+
+#if !defined(Y_PURE_FUNCTION)
+#define Y_PURE_FUNCTION
+#endif
+
+/**
+ * @ def Y_HAVE_INT128
+ *
+ * Defined when the compiler supports __int128 extension
+ *
+ * @code
+ *
+ * #if defined(Y_HAVE_INT128)
+ * __int128 myVeryBigInt = 12345678901234567890;
+ * #endif
+ *
+ * @endcode
+ */
+#if defined(__SIZEOF_INT128__)
+#define Y_HAVE_INT128 1
+#endif
+
+/**
+ * XRAY macro must be passed to compiler if XRay is enabled.
+ *
+ * Define everything XRay-specific as a macro so that it doesn't cause errors
+ * for compilers that doesn't support XRay.
+ */
+#if defined(XRAY) && defined(__cplusplus)
+#include
+#define Y_XRAY_ALWAYS_INSTRUMENT [[clang::xray_always_instrument]]
+#define Y_XRAY_NEVER_INSTRUMENT [[clang::xray_never_instrument]]
+#define Y_XRAY_CUSTOM_EVENT(__string, __length) \
+ do { \
+ __xray_customevent(__string, __length); \
+ } while (0)
+#else
+#define Y_XRAY_ALWAYS_INSTRUMENT
+#define Y_XRAY_NEVER_INSTRUMENT
+#define Y_XRAY_CUSTOM_EVENT(__string, __length) \
+ do { \
+ } while (0)
+#endif
diff --git a/contrib/lfalloc/src/util/system/defaults.h b/contrib/lfalloc/src/util/system/defaults.h
new file mode 100644
index 00000000000..19196a28b2b
--- /dev/null
+++ b/contrib/lfalloc/src/util/system/defaults.h
@@ -0,0 +1,168 @@
+#pragma once
+
+#include "platform.h"
+
+#if defined _unix_
+#define LOCSLASH_C '/'
+#define LOCSLASH_S "/"
+#else
+#define LOCSLASH_C '\\'
+#define LOCSLASH_S "\\"
+#endif // _unix_
+
+#if defined(__INTEL_COMPILER) && defined(__cplusplus)
+#include
+#endif
+
+// low and high parts of integers
+#if !defined(_win_)
+#include
+#endif
+
+#if defined(BSD) || defined(_android_)
+
+#if defined(BSD)
+#include
+#endif
+
+#if defined(_android_)
+#include
+#endif
+
+#if (BYTE_ORDER == LITTLE_ENDIAN)
+#define _little_endian_
+#elif (BYTE_ORDER == BIG_ENDIAN)
+#define _big_endian_
+#else
+#error unknown endian not supported
+#endif
+
+#elif (defined(_sun_) && !defined(__i386__)) || defined(_hpux_) || defined(WHATEVER_THAT_HAS_BIG_ENDIAN)
+#define _big_endian_
+#else
+#define _little_endian_
+#endif
+
+// alignment
+#if (defined(_sun_) && !defined(__i386__)) || defined(_hpux_) || defined(__alpha__) || defined(__ia64__) || defined(WHATEVER_THAT_NEEDS_ALIGNING_QUADS)
+#define _must_align8_
+#endif
+
+#if (defined(_sun_) && !defined(__i386__)) || defined(_hpux_) || defined(__alpha__) || defined(__ia64__) || defined(WHATEVER_THAT_NEEDS_ALIGNING_LONGS)
+#define _must_align4_
+#endif
+
+#if (defined(_sun_) && !defined(__i386__)) || defined(_hpux_) || defined(__alpha__) || defined(__ia64__) || defined(WHATEVER_THAT_NEEDS_ALIGNING_SHORTS)
+#define _must_align2_
+#endif
+
+#if defined(__GNUC__)
+#define alias_hack __attribute__((__may_alias__))
+#endif
+
+#ifndef alias_hack
+#define alias_hack
+#endif
+
+#include "types.h"
+
+#if defined(__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L)
+#define PRAGMA(x) _Pragma(#x)
+#define RCSID(idstr) PRAGMA(comment(exestr, idstr))
+#else
+#define RCSID(idstr) static const char rcsid[] = idstr
+#endif
+
+#include "compiler.h"
+
+#ifdef _win_
+#include
+#elif defined(_sun_)
+#include
+#endif
+
+#ifdef NDEBUG
+#define Y_IF_DEBUG(X)
+#else
+#define Y_IF_DEBUG(X) X
+#endif
+
+/**
+ * @def Y_ARRAY_SIZE
+ *
+ * This macro is needed to get number of elements in a statically allocated fixed size array. The
+ * expression is a compile-time constant and therefore can be used in compile time computations.
+ *
+ * @code
+ * enum ENumbers {
+ * EN_ONE,
+ * EN_TWO,
+ * EN_SIZE
+ * }
+ *
+ * const char* NAMES[] = {
+ * "one",
+ * "two"
+ * }
+ *
+ * static_assert(Y_ARRAY_SIZE(NAMES) == EN_SIZE, "you should define `NAME` for each enumeration");
+ * @endcode
+ *
+ * This macro also catches type errors. If you see a compiler error like "warning: division by zero
+ * is undefined" when using `Y_ARRAY_SIZE` then you are probably giving it a pointer.
+ *
+ * Since all of our code is expected to work on a 64 bit platform where pointers are 8 bytes we may
+ * falsefully accept pointers to types of sizes that are divisors of 8 (1, 2, 4 and 8).
+ */
+#if defined(__cplusplus)
+namespace NArraySizePrivate {
+ template
+ struct TArraySize;
+
+ template
+ struct TArraySize {
+ enum {
+ Result = N
+ };
+ };
+
+ template
+ struct TArraySize {
+ enum {
+ Result = N
+ };
+ };
+}
+
+#define Y_ARRAY_SIZE(arr) ((size_t)::NArraySizePrivate::TArraySize::Result)
+#else
+#undef Y_ARRAY_SIZE
+#define Y_ARRAY_SIZE(arr) \
+ ((sizeof(arr) / sizeof((arr)[0])) / static_cast(!(sizeof(arr) % sizeof((arr)[0]))))
+#endif
+
+#undef Y_ARRAY_BEGIN
+#define Y_ARRAY_BEGIN(arr) (arr)
+
+#undef Y_ARRAY_END
+#define Y_ARRAY_END(arr) ((arr) + Y_ARRAY_SIZE(arr))
+
+/**
+ * Concatenates two symbols, even if one of them is itself a macro.
+ */
+#define Y_CAT(X, Y) Y_CAT_I(X, Y)
+#define Y_CAT_I(X, Y) Y_CAT_II(X, Y)
+#define Y_CAT_II(X, Y) X##Y
+
+#define Y_STRINGIZE(X) UTIL_PRIVATE_STRINGIZE_AUX(X)
+#define UTIL_PRIVATE_STRINGIZE_AUX(X) #X
+
+#if defined(__COUNTER__)
+#define Y_GENERATE_UNIQUE_ID(N) Y_CAT(N, __COUNTER__)
+#endif
+
+#if !defined(Y_GENERATE_UNIQUE_ID)
+#define Y_GENERATE_UNIQUE_ID(N) Y_CAT(N, __LINE__)
+#endif
+
+#define NPOS ((size_t)-1)
diff --git a/contrib/lfalloc/src/util/system/platform.h b/contrib/lfalloc/src/util/system/platform.h
new file mode 100644
index 00000000000..0687f239a2e
--- /dev/null
+++ b/contrib/lfalloc/src/util/system/platform.h
@@ -0,0 +1,242 @@
+#pragma once
+
+// What OS ?
+// our definition has the form _{osname}_
+
+#if defined(_WIN64)
+#define _win64_
+#define _win32_
+#elif defined(__WIN32__) || defined(_WIN32) // _WIN32 is also defined by the 64-bit compiler for backward compatibility
+#define _win32_
+#else
+#define _unix_
+#if defined(__sun__) || defined(sun) || defined(sparc) || defined(__sparc)
+#define _sun_
+#endif
+#if defined(__hpux__)
+#define _hpux_
+#endif
+#if defined(__linux__)
+#define _linux_
+#endif
+#if defined(__FreeBSD__)
+#define _freebsd_
+#endif
+#if defined(__CYGWIN__)
+#define _cygwin_
+#endif
+#if defined(__APPLE__)
+#define _darwin_
+#endif
+#if defined(__ANDROID__)
+#define _android_
+#endif
+#endif
+
+#if defined(__IOS__)
+#define _ios_
+#endif
+
+#if defined(_linux_)
+#if defined(_musl_)
+//nothing to do
+#elif defined(_android_)
+#define _bionic_
+#else
+#define _glibc_
+#endif
+#endif
+
+#if defined(_darwin_)
+#define unix
+#define __unix__
+#endif
+
+#if defined(_win32_) || defined(_win64_)
+#define _win_
+#endif
+
+#if defined(__arm__) || defined(__ARM__) || defined(__ARM_NEON) || defined(__aarch64__) || defined(_M_ARM)
+#if defined(__arm64) || defined(__arm64__) || defined(__aarch64__)
+#define _arm64_
+#else
+#define _arm32_
+#endif
+#endif
+
+#if defined(_arm64_) || defined(_arm32_)
+#define _arm_
+#endif
+
+/* __ia64__ and __x86_64__ - defined by GNU C.
+ * _M_IA64, _M_X64, _M_AMD64 - defined by Visual Studio.
+ *
+ * Microsoft can define _M_IX86, _M_AMD64 (before Visual Studio 8)
+ * or _M_X64 (starting in Visual Studio 8).
+ */
+#if defined(__x86_64__) || defined(_M_X64) || defined(_M_AMD64)
+#define _x86_64_
+#endif
+
+#if defined(__i386__) || defined(_M_IX86)
+#define _i386_
+#endif
+
+#if defined(__ia64__) || defined(_M_IA64)
+#define _ia64_
+#endif
+
+#if defined(__powerpc__)
+#define _ppc_
+#endif
+
+#if defined(__powerpc64__)
+#define _ppc64_
+#endif
+
+#if !defined(sparc) && !defined(__sparc) && !defined(__hpux__) && !defined(__alpha__) && !defined(_ia64_) && !defined(_x86_64_) && !defined(_arm_) && !defined(_i386_) && !defined(_ppc_) && !defined(_ppc64_)
+#error "platform not defined, please, define one"
+#endif
+
+#if defined(_x86_64_) || defined(_i386_)
+#define _x86_
+#endif
+
+#if defined(__MIC__)
+#define _mic_
+#define _k1om_
+#endif
+
+// stdio or MessageBox
+#if defined(__CONSOLE__) || defined(_CONSOLE)
+#define _console_
+#endif
+#if (defined(_win_) && !defined(_console_))
+#define _windows_
+#elif !defined(_console_)
+#define _console_
+#endif
+
+#if defined(__SSE__) || defined(SSE_ENABLED)
+#define _sse_
+#endif
+
+#if defined(__SSE2__) || defined(SSE2_ENABLED)
+#define _sse2_
+#endif
+
+#if defined(__SSE3__) || defined(SSE3_ENABLED)
+#define _sse3_
+#endif
+
+#if defined(__SSSE3__) || defined(SSSE3_ENABLED)
+#define _ssse3_
+#endif
+
+#if defined(POPCNT_ENABLED)
+#define _popcnt_
+#endif
+
+#if defined(__DLL__) || defined(_DLL)
+#define _dll_
+#endif
+
+// 16, 32 or 64
+#if defined(__sparc_v9__) || defined(_x86_64_) || defined(_ia64_) || defined(_arm64_) || defined(_ppc64_)
+#define _64_
+#else
+#define _32_
+#endif
+
+/* All modern 64-bit Unix systems use scheme LP64 (long, pointers are 64-bit).
+ * Microsoft uses a different scheme: LLP64 (long long, pointers are 64-bit).
+ *
+ * Scheme LP64 LLP64
+ * char 8 8
+ * short 16 16
+ * int 32 32
+ * long 64 32
+ * long long 64 64
+ * pointer 64 64
+ */
+
+#if defined(_32_)
+#define SIZEOF_PTR 4
+#elif defined(_64_)
+#define SIZEOF_PTR 8
+#endif
+
+#define PLATFORM_DATA_ALIGN SIZEOF_PTR
+
+#if !defined(SIZEOF_PTR)
+#error todo
+#endif
+
+#define SIZEOF_CHAR 1
+#define SIZEOF_UNSIGNED_CHAR 1
+#define SIZEOF_SHORT 2
+#define SIZEOF_UNSIGNED_SHORT 2
+#define SIZEOF_INT 4
+#define SIZEOF_UNSIGNED_INT 4
+
+#if defined(_32_)
+#define SIZEOF_LONG 4
+#define SIZEOF_UNSIGNED_LONG 4
+#elif defined(_64_)
+#if defined(_win_)
+#define SIZEOF_LONG 4
+#define SIZEOF_UNSIGNED_LONG 4
+#else
+#define SIZEOF_LONG 8
+#define SIZEOF_UNSIGNED_LONG 8
+#endif // _win_
+#endif // _32_
+
+#if !defined(SIZEOF_LONG)
+#error todo
+#endif
+
+#define SIZEOF_LONG_LONG 8
+#define SIZEOF_UNSIGNED_LONG_LONG 8
+
+#undef SIZEOF_SIZE_T // in case we include which defines it, too
+#define SIZEOF_SIZE_T SIZEOF_PTR
+
+#if defined(__INTEL_COMPILER)
+#pragma warning(disable 1292)
+#pragma warning(disable 1469)
+#pragma warning(disable 193)
+#pragma warning(disable 271)
+#pragma warning(disable 383)
+#pragma warning(disable 424)
+#pragma warning(disable 444)
+#pragma warning(disable 584)
+#pragma warning(disable 593)
+#pragma warning(disable 981)
+#pragma warning(disable 1418)
+#pragma warning(disable 304)
+#pragma warning(disable 810)
+#pragma warning(disable 1029)
+#pragma warning(disable 1419)
+#pragma warning(disable 177)
+#pragma warning(disable 522)
+#pragma warning(disable 858)
+#pragma warning(disable 111)
+#pragma warning(disable 1599)
+#pragma warning(disable 411)
+#pragma warning(disable 304)
+#pragma warning(disable 858)
+#pragma warning(disable 444)
+#pragma warning(disable 913)
+#pragma warning(disable 310)
+#pragma warning(disable 167)
+#pragma warning(disable 180)
+#pragma warning(disable 1572)
+#endif
+
+#if defined(_MSC_VER)
+#undef _WINSOCKAPI_
+#define _WINSOCKAPI_
+#undef NOMINMAX
+#define NOMINMAX
+#endif
diff --git a/contrib/lfalloc/src/util/system/types.h b/contrib/lfalloc/src/util/system/types.h
new file mode 100644
index 00000000000..af4f0adb13d
--- /dev/null
+++ b/contrib/lfalloc/src/util/system/types.h
@@ -0,0 +1,117 @@
+#pragma once
+
+// DO_NOT_STYLE
+
+#include "platform.h"
+
+#include
+
+typedef int8_t i8;
+typedef int16_t i16;
+typedef uint8_t ui8;
+typedef uint16_t ui16;
+
+typedef int yssize_t;
+#define PRIYSZT "d"
+
+#if defined(_darwin_) && defined(_32_)
+typedef unsigned long ui32;
+typedef long i32;
+#else
+typedef uint32_t ui32;
+typedef int32_t i32;
+#endif
+
+#if defined(_darwin_) && defined(_64_)
+typedef unsigned long ui64;
+typedef long i64;
+#else
+typedef uint64_t ui64;
+typedef int64_t i64;
+#endif
+
+#define LL(number) INT64_C(number)
+#define ULL(number) UINT64_C(number)
+
+// Macro for size_t and ptrdiff_t types
+#if defined(_32_)
+# if defined(_darwin_)
+# define PRISZT "lu"
+# undef PRIi32
+# define PRIi32 "li"
+# undef SCNi32
+# define SCNi32 "li"
+# undef PRId32
+# define PRId32 "li"
+# undef SCNd32
+# define SCNd32 "li"
+# undef PRIu32
+# define PRIu32 "lu"
+# undef SCNu32
+# define SCNu32 "lu"
+# undef PRIx32
+# define PRIx32 "lx"
+# undef SCNx32
+# define SCNx32 "lx"
+# elif !defined(_cygwin_)
+# define PRISZT PRIu32
+# else
+# define PRISZT "u"
+# endif
+# define SCNSZT SCNu32
+# define PRIPDT PRIi32
+# define SCNPDT SCNi32
+# define PRITMT PRIi32
+# define SCNTMT SCNi32
+#elif defined(_64_)
+# if defined(_darwin_)
+# define PRISZT "lu"
+# undef PRIu64
+# define PRIu64 PRISZT
+# undef PRIx64
+# define PRIx64 "lx"
+# undef PRIX64
+# define PRIX64 "lX"
+# undef PRId64
+# define PRId64 "ld"
+# undef PRIi64
+# define PRIi64 "li"
+# undef SCNi64
+# define SCNi64 "li"
+# undef SCNu64
+# define SCNu64 "lu"
+# undef SCNx64
+# define SCNx64 "lx"
+# else
+# define PRISZT PRIu64
+# endif
+# define SCNSZT SCNu64
+# define PRIPDT PRIi64
+# define SCNPDT SCNi64
+# define PRITMT PRIi64
+# define SCNTMT SCNi64
+#else
+# error "Unsupported platform"
+#endif
+
+// SUPERLONG
+#if !defined(DONT_USE_SUPERLONG) && !defined(SUPERLONG_MAX)
+#define SUPERLONG_MAX ~LL(0)
+typedef i64 SUPERLONG;
+#endif
+
+// UNICODE
+// UCS-2, native byteorder
+typedef ui16 wchar16;
+// internal symbol type: UTF-16LE
+typedef wchar16 TChar;
+typedef ui32 wchar32;
+
+#if defined(_MSC_VER)
+#include
+typedef SSIZE_T ssize_t;
+#define HAVE_SSIZE_T 1
+#include
+#endif
+
+#include
diff --git a/contrib/libhdfs3-cmake/CMakeLists.txt b/contrib/libhdfs3-cmake/CMakeLists.txt
index f651e55fb7b..0d075922f07 100644
--- a/contrib/libhdfs3-cmake/CMakeLists.txt
+++ b/contrib/libhdfs3-cmake/CMakeLists.txt
@@ -208,7 +208,8 @@ target_link_libraries(hdfs3 ${LIBXML2_LIBRARY})
# inherit from parent cmake
target_include_directories(hdfs3 PRIVATE ${Boost_INCLUDE_DIRS})
target_include_directories(hdfs3 PRIVATE ${Protobuf_INCLUDE_DIR})
-target_include_directories(hdfs3 PRIVATE ${OPENSSL_INCLUDE_DIR})
-
target_link_libraries(hdfs3 ${Protobuf_LIBRARY})
-target_link_libraries(hdfs3 ${OPENSSL_LIBRARIES})
+if(OPENSSL_INCLUDE_DIR AND OPENSSL_LIBRARIES)
+ target_include_directories(hdfs3 PRIVATE ${OPENSSL_INCLUDE_DIR})
+ target_link_libraries(hdfs3 ${OPENSSL_LIBRARIES})
+endif()
diff --git a/contrib/libmetrohash/src/platform.h b/contrib/libmetrohash/src/platform.h
index 31291b94b33..bc00e5a286b 100644
--- a/contrib/libmetrohash/src/platform.h
+++ b/contrib/libmetrohash/src/platform.h
@@ -18,6 +18,7 @@
#define METROHASH_PLATFORM_H
#include
+#include
// rotate right idiom recognized by most compilers
inline static uint64_t rotate_right(uint64_t v, unsigned k)
@@ -25,20 +26,28 @@ inline static uint64_t rotate_right(uint64_t v, unsigned k)
return (v >> k) | (v << (64 - k));
}
-// unaligned reads, fast and safe on Nehalem and later microarchitectures
inline static uint64_t read_u64(const void * const ptr)
{
- return static_cast(*reinterpret_cast(ptr));
+ uint64_t result;
+ // Assignment like `result = *reinterpret_cast(ptr)` here would mean undefined behaviour (unaligned read),
+ // so we use memcpy() which is the most portable. clang & gcc usually translates `memcpy()` into a single `load` instruction
+ // when hardware supports it, so using memcpy() is efficient too.
+ memcpy(&result, ptr, sizeof(result));
+ return result;
}
inline static uint64_t read_u32(const void * const ptr)
{
- return static_cast(*reinterpret_cast(ptr));
+ uint32_t result;
+ memcpy(&result, ptr, sizeof(result));
+ return result;
}
inline static uint64_t read_u16(const void * const ptr)
{
- return static_cast(*reinterpret_cast(ptr));
+ uint16_t result;
+ memcpy(&result, ptr, sizeof(result));
+ return result;
}
inline static uint64_t read_u8 (const void * const ptr)
diff --git a/contrib/librdkafka b/contrib/librdkafka
index 73295a702cd..8695b9d63ac 160000
--- a/contrib/librdkafka
+++ b/contrib/librdkafka
@@ -1 +1 @@
-Subproject commit 73295a702cd1c85c11749ade500d713db7099cca
+Subproject commit 8695b9d63ac0fe1b891b511d5b36302ffc84d4e2
diff --git a/contrib/librdkafka-cmake/CMakeLists.txt b/contrib/librdkafka-cmake/CMakeLists.txt
index 8cc49ce0c17..27e3a8cec05 100644
--- a/contrib/librdkafka-cmake/CMakeLists.txt
+++ b/contrib/librdkafka-cmake/CMakeLists.txt
@@ -58,4 +58,7 @@ add_library(rdkafka ${LINK_MODE} ${SRCS})
target_include_directories(rdkafka SYSTEM PUBLIC include)
target_include_directories(rdkafka SYSTEM PUBLIC ${RDKAFKA_SOURCE_DIR}) # Because weird logic with "include_next" is used.
target_include_directories(rdkafka SYSTEM PRIVATE ${ZSTD_INCLUDE_DIR}/common) # Because wrong path to "zstd_errors.h" is used.
-target_link_libraries(rdkafka PUBLIC ${ZLIB_LIBRARIES} ${ZSTD_LIBRARY} ${LZ4_LIBRARY} ${OPENSSL_SSL_LIBRARY} ${OPENSSL_CRYPTO_LIBRARY})
+target_link_libraries(rdkafka PUBLIC ${ZLIB_LIBRARIES} ${ZSTD_LIBRARY} ${LZ4_LIBRARY})
+if(OPENSSL_SSL_LIBRARY AND OPENSSL_CRYPTO_LIBRARY)
+ target_link_libraries(rdkafka PUBLIC ${OPENSSL_SSL_LIBRARY} ${OPENSSL_CRYPTO_LIBRARY})
+endif()
diff --git a/contrib/lz4 b/contrib/lz4
index c10863b98e1..780aac520b6 160000
--- a/contrib/lz4
+++ b/contrib/lz4
@@ -1 +1 @@
-Subproject commit c10863b98e1503af90616ae99725ecd120265dfb
+Subproject commit 780aac520b69d6369f4e3995624c37e56d75498d
diff --git a/contrib/lz4-cmake/CMakeLists.txt b/contrib/lz4-cmake/CMakeLists.txt
index 382185cc339..25cceaa4574 100644
--- a/contrib/lz4-cmake/CMakeLists.txt
+++ b/contrib/lz4-cmake/CMakeLists.txt
@@ -9,8 +9,7 @@ add_library (lz4
${LIBRARY_DIR}/xxhash.h
${LIBRARY_DIR}/lz4.h
- ${LIBRARY_DIR}/lz4hc.h
- ${LIBRARY_DIR}/lz4opt.h)
+ ${LIBRARY_DIR}/lz4hc.h)
target_compile_definitions(lz4 PUBLIC LZ4_DISABLE_DEPRECATE_WARNINGS=1)
diff --git a/contrib/mariadb-connector-c-cmake/CMakeLists.txt b/contrib/mariadb-connector-c-cmake/CMakeLists.txt
index 4c1184b3edb..9b095a2e15b 100644
--- a/contrib/mariadb-connector-c-cmake/CMakeLists.txt
+++ b/contrib/mariadb-connector-c-cmake/CMakeLists.txt
@@ -33,7 +33,6 @@ ${MARIADB_CLIENT_SOURCE_DIR}/libmariadb/ma_time.c
${MARIADB_CLIENT_SOURCE_DIR}/libmariadb/ma_tls.c
#${MARIADB_CLIENT_SOURCE_DIR}/libmariadb/secure/gnutls.c
#${MARIADB_CLIENT_SOURCE_DIR}/libmariadb/secure/ma_schannel.c
-${MARIADB_CLIENT_SOURCE_DIR}/libmariadb/secure/openssl.c
#${MARIADB_CLIENT_SOURCE_DIR}/libmariadb/secure/schannel.c
#${MARIADB_CLIENT_SOURCE_DIR}/plugins/auth/auth_gssapi_client.c
#${MARIADB_CLIENT_SOURCE_DIR}/plugins/auth/dialog.c
@@ -55,12 +54,19 @@ ${MARIADB_CLIENT_SOURCE_DIR}/plugins/pvio/pvio_socket.c
${CMAKE_CURRENT_SOURCE_DIR}/linux_x86_64/libmariadb/ma_client_plugin.c
)
+if(OPENSSL_LIBRARIES)
+ list(APPEND SRCS ${MARIADB_CLIENT_SOURCE_DIR}/libmariadb/secure/openssl.c)
+endif()
+
add_library(mysqlclient STATIC ${SRCS})
-target_link_libraries(mysqlclient ${OPENSSL_LIBRARIES})
+if(OPENSSL_LIBRARIES)
+ target_link_libraries(mysqlclient ${OPENSSL_LIBRARIES})
+ target_compile_definitions(mysqlclient PRIVATE -D HAVE_OPENSSL -D HAVE_TLS)
+endif()
target_include_directories(mysqlclient PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/linux_x86_64/include)
target_include_directories(mysqlclient PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/common/include)
target_include_directories(mysqlclient PUBLIC ${MARIADB_CLIENT_SOURCE_DIR}/include)
-target_compile_definitions(mysqlclient PRIVATE -D THREAD -D HAVE_OPENSSL -D HAVE_TLS)
+target_compile_definitions(mysqlclient PRIVATE -D THREAD)
diff --git a/contrib/poco b/contrib/poco
index fe5505e56c2..29439cf7fa3 160000
--- a/contrib/poco
+++ b/contrib/poco
@@ -1 +1 @@
-Subproject commit fe5505e56c27b6ecb0dcbc40c49dc2caf4e9637f
+Subproject commit 29439cf7fa32c1a2d62d925bb6d6a3f14668a4a2
diff --git a/dbms/CMakeLists.txt b/dbms/CMakeLists.txt
index a07d658c4e6..63e97f4e061 100644
--- a/dbms/CMakeLists.txt
+++ b/dbms/CMakeLists.txt
@@ -20,7 +20,7 @@ set (CONFIG_VERSION ${CMAKE_CURRENT_BINARY_DIR}/src/Common/config_version.h)
set (CONFIG_COMMON ${CMAKE_CURRENT_BINARY_DIR}/src/Common/config.h)
include (cmake/version.cmake)
-message (STATUS "Will build ${VERSION_FULL} revision ${VERSION_REVISION}")
+message (STATUS "Will build ${VERSION_FULL} revision ${VERSION_REVISION} ${VERSION_OFFICIAL}")
configure_file (src/Common/config.h.in ${CONFIG_COMMON})
configure_file (src/Common/config_version.h.in ${CONFIG_VERSION})
@@ -57,7 +57,7 @@ if (CMAKE_CXX_COMPILER_ID STREQUAL "Clang")
endif ()
if (NOT CMAKE_CXX_COMPILER_VERSION VERSION_LESS 8)
- set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wextra-semi-stmt -Wshadow-field -Wstring-plus-int")
+ set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wextra-semi-stmt -Wshadow-field -Wstring-plus-int -Wempty-init-stmt")
endif ()
if (NOT CMAKE_CXX_COMPILER_VERSION VERSION_LESS 9)
@@ -155,7 +155,6 @@ if (USE_EMBEDDED_COMPILER)
target_include_directories (dbms SYSTEM BEFORE PUBLIC ${LLVM_INCLUDE_DIRS})
endif ()
-
if (CMAKE_BUILD_TYPE_UC STREQUAL "RELEASE" OR CMAKE_BUILD_TYPE_UC STREQUAL "RELWITHDEBINFO" OR CMAKE_BUILD_TYPE_UC STREQUAL "MINSIZEREL")
# Won't generate debug info for files with heavy template instantiation to achieve faster linking and lower size.
set_source_files_properties(
@@ -186,8 +185,6 @@ target_link_libraries (clickhouse_common_io
${LINK_LIBRARIES_ONLY_ON_X86_64}
PUBLIC
${DOUBLE_CONVERSION_LIBRARIES}
- PRIVATE
- pocoext
PUBLIC
${Poco_Net_LIBRARY}
${Poco_Util_LIBRARY}
@@ -197,8 +194,7 @@ target_link_libraries (clickhouse_common_io
${CITYHASH_LIBRARIES}
PRIVATE
${ZLIB_LIBRARIES}
- ${EXECINFO_LIBRARY}
- ${ELF_LIBRARY}
+ ${EXECINFO_LIBRARIES}
PUBLIC
${Boost_SYSTEM_LIBRARY}
PRIVATE
@@ -214,6 +210,10 @@ target_link_libraries (clickhouse_common_io
target_include_directories(clickhouse_common_io SYSTEM BEFORE PUBLIC ${RE2_INCLUDE_DIR})
+if (USE_LFALLOC)
+ target_include_directories (clickhouse_common_io SYSTEM BEFORE PUBLIC ${LFALLOC_INCLUDE_DIR})
+endif ()
+
if(CPUID_LIBRARY)
target_link_libraries(clickhouse_common_io PRIVATE ${CPUID_LIBRARY})
endif()
@@ -223,8 +223,9 @@ if(CPUINFO_LIBRARY)
endif()
target_link_libraries (dbms
- PRIVATE
+ PUBLIC
clickhouse_compression
+ PRIVATE
clickhouse_parsers
clickhouse_common_config
PUBLIC
@@ -232,7 +233,6 @@ target_link_libraries (dbms
PRIVATE
clickhouse_dictionaries_embedded
PUBLIC
- pocoext
${MYSQLXX_LIBRARY}
PRIVATE
${BTRIE_LIBRARIES}
@@ -309,7 +309,10 @@ if (USE_PARQUET)
endif ()
endif ()
-target_link_libraries(dbms PRIVATE ${OPENSSL_CRYPTO_LIBRARY} Threads::Threads)
+if(OPENSSL_CRYPTO_LIBRARY)
+ target_link_libraries(dbms PRIVATE ${OPENSSL_CRYPTO_LIBRARY})
+endif ()
+target_link_libraries(dbms PRIVATE Threads::Threads)
target_include_directories (dbms SYSTEM BEFORE PRIVATE ${DIVIDE_INCLUDE_DIR})
target_include_directories (dbms SYSTEM BEFORE PRIVATE ${SPARCEHASH_INCLUDE_DIR})
diff --git a/dbms/cmake/version.cmake b/dbms/cmake/version.cmake
index 7df40c7c0d4..3557d3d97df 100644
--- a/dbms/cmake/version.cmake
+++ b/dbms/cmake/version.cmake
@@ -1,11 +1,11 @@
# This strings autochanged from release_lib.sh:
-set(VERSION_REVISION 54417)
+set(VERSION_REVISION 54419)
set(VERSION_MAJOR 19)
-set(VERSION_MINOR 5)
+set(VERSION_MINOR 7)
set(VERSION_PATCH 1)
-set(VERSION_GITHASH 628ed349c335b79a441a1bd6e4bc791d61dfe62c)
-set(VERSION_DESCRIBE v19.5.1.1-testing)
-set(VERSION_STRING 19.5.1.1)
+set(VERSION_GITHASH b0b369b30f04a5026d1da5c7d3fd5998d6de1fe4)
+set(VERSION_DESCRIBE v19.7.1.1-testing)
+set(VERSION_STRING 19.7.1.1)
# end of autochange
set(VERSION_EXTRA "" CACHE STRING "")
@@ -24,3 +24,7 @@ set (VERSION_FULL "${VERSION_NAME} ${VERSION_STRING}")
set (VERSION_SO "${VERSION_STRING}")
math (EXPR VERSION_INTEGER "${VERSION_PATCH} + ${VERSION_MINOR}*1000 + ${VERSION_MAJOR}*1000000")
+
+if(YANDEX_OFFICIAL_BUILD)
+ set(VERSION_OFFICIAL " (official build)")
+endif()
diff --git a/dbms/programs/CMakeLists.txt b/dbms/programs/CMakeLists.txt
index be878cce1ab..b6928652801 100644
--- a/dbms/programs/CMakeLists.txt
+++ b/dbms/programs/CMakeLists.txt
@@ -93,6 +93,7 @@ if (CLICKHOUSE_ONE_SHARED)
target_link_libraries(clickhouse-lib ${CLICKHOUSE_SERVER_LINK} ${CLICKHOUSE_CLIENT_LINK} ${CLICKHOUSE_LOCAL_LINK} ${CLICKHOUSE_BENCHMARK_LINK} ${CLICKHOUSE_PERFORMANCE_TEST_LINK} ${CLICKHOUSE_COPIER_LINK} ${CLICKHOUSE_EXTRACT_FROM_CONFIG_LINK} ${CLICKHOUSE_COMPRESSOR_LINK} ${CLICKHOUSE_FORMAT_LINK} ${CLICKHOUSE_OBFUSCATOR_LINK} ${CLICKHOUSE_COMPILER_LINK} ${CLICKHOUSE_ODBC_BRIDGE_LINK})
target_include_directories(clickhouse-lib ${CLICKHOUSE_SERVER_INCLUDE} ${CLICKHOUSE_CLIENT_INCLUDE} ${CLICKHOUSE_LOCAL_INCLUDE} ${CLICKHOUSE_BENCHMARK_INCLUDE} ${CLICKHOUSE_PERFORMANCE_TEST_INCLUDE} ${CLICKHOUSE_COPIER_INCLUDE} ${CLICKHOUSE_EXTRACT_FROM_CONFIG_INCLUDE} ${CLICKHOUSE_COMPRESSOR_INCLUDE} ${CLICKHOUSE_FORMAT_INCLUDE} ${CLICKHOUSE_OBFUSCATOR_INCLUDE} ${CLICKHOUSE_COMPILER_INCLUDE} ${CLICKHOUSE_ODBC_BRIDGE_INCLUDE})
set_target_properties(clickhouse-lib PROPERTIES SOVERSION ${VERSION_MAJOR}.${VERSION_MINOR} VERSION ${VERSION_SO} OUTPUT_NAME clickhouse DEBUG_POSTFIX "")
+ install (TARGETS clickhouse-lib LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR} COMPONENT clickhouse)
endif()
if (CLICKHOUSE_SPLIT_BINARY)
@@ -154,7 +155,7 @@ else ()
clickhouse_target_link_split_lib(clickhouse obfuscator)
endif ()
if (USE_EMBEDDED_COMPILER)
- clickhouse_target_link_split_lib(clickhouse compiler)
+ target_link_libraries(clickhouse PRIVATE clickhouse-compiler-lib)
endif ()
set (CLICKHOUSE_BUNDLE)
diff --git a/dbms/programs/benchmark/Benchmark.cpp b/dbms/programs/benchmark/Benchmark.cpp
index b366add0ba5..89b363a2709 100644
--- a/dbms/programs/benchmark/Benchmark.cpp
+++ b/dbms/programs/benchmark/Benchmark.cpp
@@ -439,7 +439,7 @@ int mainEntryClickHouseBenchmark(int argc, char ** argv)
("help", "produce help message")
("concurrency,c", value()->default_value(1), "number of parallel queries")
("delay,d", value()->default_value(1), "delay between intermediate reports in seconds (set 0 to disable reports)")
- ("stage", value()->default_value("complete"), "request query processing up to specified stage")
+ ("stage", value()->default_value("complete"), "request query processing up to specified stage: complete,fetch_columns,with_mergeable_state")
("iterations,i", value()->default_value(0), "amount of queries to be executed")
("timelimit,t", value()->default_value(0.), "stop launch of queries after specified time limit")
("randomize,r", value()->default_value(false), "randomize order of execution")
diff --git a/dbms/programs/clang/Compiler-5.0.0/CMakeLists.txt b/dbms/programs/clang/Compiler-5.0.0/CMakeLists.txt
index 7fe0cd92ef7..e0171630bf2 100644
--- a/dbms/programs/clang/Compiler-5.0.0/CMakeLists.txt
+++ b/dbms/programs/clang/Compiler-5.0.0/CMakeLists.txt
@@ -46,7 +46,7 @@ LLVMSupport
#PollyISL
#PollyPPCG
-PUBLIC ${ZLIB_LIBRARIES} ${EXECINFO_LIBRARY} Threads::Threads
+PUBLIC ${ZLIB_LIBRARIES} ${EXECINFO_LIBRARIES} Threads::Threads
${MALLOC_LIBRARIES}
${GLIBC_COMPATIBILITY_LIBRARIES}
${MEMCPY_LIBRARIES}
diff --git a/dbms/programs/clang/Compiler-6.0.0/CMakeLists.txt b/dbms/programs/clang/Compiler-6.0.0/CMakeLists.txt
index b96bdb0647a..bac622ab09e 100644
--- a/dbms/programs/clang/Compiler-6.0.0/CMakeLists.txt
+++ b/dbms/programs/clang/Compiler-6.0.0/CMakeLists.txt
@@ -46,7 +46,7 @@ ${REQUIRED_LLVM_LIBRARIES}
#PollyISL
#PollyPPCG
-PUBLIC ${ZLIB_LIBRARIES} ${EXECINFO_LIBRARY} Threads::Threads
+PUBLIC ${ZLIB_LIBRARIES} ${EXECINFO_LIBRARIES} Threads::Threads
${MALLOC_LIBRARIES}
${GLIBC_COMPATIBILITY_LIBRARIES}
${MEMCPY_LIBRARIES}
diff --git a/dbms/programs/clang/Compiler-7.0.0/CMakeLists.txt b/dbms/programs/clang/Compiler-7.0.0/CMakeLists.txt
index 8b6ba6be994..35e23cc6b46 100644
--- a/dbms/programs/clang/Compiler-7.0.0/CMakeLists.txt
+++ b/dbms/programs/clang/Compiler-7.0.0/CMakeLists.txt
@@ -42,7 +42,7 @@ lldCore
${REQUIRED_LLVM_LIBRARIES}
-PUBLIC ${ZLIB_LIBRARIES} ${EXECINFO_LIBRARY} Threads::Threads
+PUBLIC ${ZLIB_LIBRARIES} ${EXECINFO_LIBRARIES} Threads::Threads
${MALLOC_LIBRARIES}
${GLIBC_COMPATIBILITY_LIBRARIES}
${MEMCPY_LIBRARIES}
diff --git a/dbms/programs/clang/Compiler-7.0.0bundled/CMakeLists.txt b/dbms/programs/clang/Compiler-7.0.0bundled/CMakeLists.txt
index d0ccc8d672c..d03052ffc28 100644
--- a/dbms/programs/clang/Compiler-7.0.0bundled/CMakeLists.txt
+++ b/dbms/programs/clang/Compiler-7.0.0bundled/CMakeLists.txt
@@ -42,7 +42,7 @@ lldCore
${REQUIRED_LLVM_LIBRARIES}
-PUBLIC ${ZLIB_LIBRARIES} ${EXECINFO_LIBRARY} Threads::Threads
+PUBLIC ${ZLIB_LIBRARIES} ${EXECINFO_LIBRARIES} Threads::Threads
${MALLOC_LIBRARIES}
${GLIBC_COMPATIBILITY_LIBRARIES}
${MEMCPY_LIBRARIES}
diff --git a/dbms/programs/clickhouse-split-helper b/dbms/programs/clickhouse-split-helper
index 2495160e02a..14a86f76097 100755
--- a/dbms/programs/clickhouse-split-helper
+++ b/dbms/programs/clickhouse-split-helper
@@ -1,5 +1,11 @@
#!/bin/sh
+# Helper for split build mode.
+# Allows to run commands like
+# clickhouse client
+# clickhouse server
+# ...
+
set -e
CMD=$1
shift
diff --git a/dbms/programs/client/Client.cpp b/dbms/programs/client/Client.cpp
index 6b4a0c6eb58..54271996290 100644
--- a/dbms/programs/client/Client.cpp
+++ b/dbms/programs/client/Client.cpp
@@ -42,6 +42,7 @@
#include
#include
#include
+#include
#include
#include
#include
@@ -101,6 +102,7 @@ namespace ErrorCodes
extern const int LOGICAL_ERROR;
extern const int CANNOT_SET_SIGNAL_HANDLER;
extern const int CANNOT_READLINE;
+ extern const int SYSTEM_ERROR;
}
@@ -295,7 +297,6 @@ private:
/// The value of the option is used as the text of query (or of multiple queries).
/// If stdin is not a terminal, INSERT data for the first query is read from it.
/// - stdin is not a terminal. In this case queries are read from it.
- stdin_is_not_tty = !isatty(STDIN_FILENO);
if (stdin_is_not_tty || config().has("query"))
is_interactive = false;
@@ -610,9 +611,6 @@ private:
try
{
- /// Determine the terminal size.
- ioctl(0, TIOCGWINSZ, &terminal_size);
-
if (!process(input))
break;
}
@@ -704,7 +702,7 @@ private:
return true;
}
- ASTInsertQuery * insert = typeid_cast(ast.get());
+ auto * insert = ast->as();
if (insert && insert->data)
{
@@ -799,22 +797,38 @@ private:
written_progress_chars = 0;
written_first_block = false;
- const ASTSetQuery * set_query = typeid_cast(&*parsed_query);
- const ASTUseQuery * use_query = typeid_cast(&*parsed_query);
- /// INSERT query for which data transfer is needed (not an INSERT SELECT) is processed separately.
- const ASTInsertQuery * insert = typeid_cast(&*parsed_query);
+ {
+ /// Temporarily apply query settings to context.
+ std::optional old_settings;
+ SCOPE_EXIT({ if (old_settings) context.setSettings(*old_settings); });
+ auto apply_query_settings = [&](const IAST & settings_ast)
+ {
+ if (!old_settings)
+ old_settings.emplace(context.getSettingsRef());
+ for (const auto & change : settings_ast.as()->changes)
+ context.setSetting(change.name, change.value);
+ };
+ const auto * insert = parsed_query->as();
+ if (insert && insert->settings_ast)
+ apply_query_settings(*insert->settings_ast);
+ /// FIXME: try to prettify this cast using `as<>()`
+ const auto * with_output = dynamic_cast(parsed_query.get());
+ if (with_output && with_output->settings_ast)
+ apply_query_settings(*with_output->settings_ast);
- connection->forceConnected();
+ connection->forceConnected();
- if (insert && !insert->select)
- processInsertQuery();
- else
- processOrdinaryQuery();
+ /// INSERT query for which data transfer is needed (not an INSERT SELECT) is processed separately.
+ if (insert && !insert->select)
+ processInsertQuery();
+ else
+ processOrdinaryQuery();
+ }
/// Do not change context (current DB, settings) in case of an exception.
if (!got_exception)
{
- if (set_query)
+ if (const auto * set_query = parsed_query->as())
{
/// Save all changes in settings to avoid losing them if the connection is lost.
for (const auto & change : set_query->changes)
@@ -826,7 +840,7 @@ private:
}
}
- if (use_query)
+ if (const auto * use_query = parsed_query->as())
{
const String & new_database = use_query->database;
/// If the client initiates the reconnection, it takes the settings from the config.
@@ -858,7 +872,7 @@ private:
/// Convert external tables to ExternalTableData and send them using the connection.
void sendExternalTables()
{
- auto * select = typeid_cast(&*parsed_query);
+ const auto * select = parsed_query->as();
if (!select && !external_tables.empty())
throw Exception("External tables could be sent only with select query", ErrorCodes::BAD_ARGUMENTS);
@@ -883,7 +897,7 @@ private:
void processInsertQuery()
{
/// Send part of query without data, because data will be sent separately.
- const ASTInsertQuery & parsed_insert_query = typeid_cast(*parsed_query);
+ const auto & parsed_insert_query = parsed_query->as();
String query_without_data = parsed_insert_query.data
? query.substr(0, parsed_insert_query.data - query.data())
: query;
@@ -940,7 +954,7 @@ private:
void sendData(Block & sample, const ColumnsDescription & columns_description)
{
/// If INSERT data must be sent.
- const ASTInsertQuery * parsed_insert_query = typeid_cast(&*parsed_query);
+ const auto * parsed_insert_query = parsed_query->as();
if (!parsed_insert_query)
return;
@@ -965,18 +979,16 @@ private:
String current_format = insert_format;
/// Data format can be specified in the INSERT query.
- if (ASTInsertQuery * insert = typeid_cast(&*parsed_query))
+ if (const auto * insert = parsed_query->as())
{
if (!insert->format.empty())
current_format = insert->format;
- if (insert->settings_ast)
- InterpreterSetQuery(insert->settings_ast, context).executeForCurrentContext();
}
BlockInputStreamPtr block_input = context.getInputFormat(
current_format, buf, sample, insert_format_max_block_size);
- const auto & column_defaults = columns_description.defaults;
+ const auto & column_defaults = columns_description.getDefaults();
if (!column_defaults.empty())
block_input = std::make_shared(block_input, column_defaults, context);
@@ -1231,12 +1243,14 @@ private:
String current_format = format;
/// The query can specify output format or output file.
- if (ASTQueryWithOutput * query_with_output = dynamic_cast(&*parsed_query))
+ /// FIXME: try to prettify this cast using `as<>()`
+ if (const auto * query_with_output = dynamic_cast(parsed_query.get()))
{
- if (query_with_output->out_file != nullptr)
+ if (query_with_output->out_file)
{
- const auto & out_file_node = typeid_cast(*query_with_output->out_file);
+ const auto & out_file_node = query_with_output->out_file->as();
const auto & out_file = out_file_node.value.safeGet();
+
out_file_buf.emplace(out_file, DBMS_DEFAULT_BUFFER_SIZE, O_WRONLY | O_EXCL | O_CREAT);
out_buf = &*out_file_buf;
@@ -1248,13 +1262,9 @@ private:
{
if (has_vertical_output_suffix)
throw Exception("Output format already specified", ErrorCodes::CLIENT_OUTPUT_FORMAT_SPECIFIED);
- const auto & id = typeid_cast(*query_with_output->format);
+ const auto & id = query_with_output->format->as();
current_format = id.name;
}
- if (query_with_output->settings_ast)
- {
- InterpreterSetQuery(query_with_output->settings_ast, context).executeForCurrentContext();
- }
}
if (has_vertical_output_suffix)
@@ -1318,6 +1328,9 @@ private:
/// Received data block is immediately displayed to the user.
block_out_stream->flush();
+
+ /// Restore progress bar after data block.
+ writeProgress();
}
@@ -1357,8 +1370,8 @@ private:
void clearProgress()
{
- std::cerr << RESTORE_CURSOR_POSITION CLEAR_TO_END_OF_LINE;
written_progress_chars = 0;
+ std::cerr << RESTORE_CURSOR_POSITION CLEAR_TO_END_OF_LINE;
}
@@ -1367,6 +1380,9 @@ private:
if (!need_render_progress)
return;
+ /// Output all progress bar commands to stderr at once to avoid flicker.
+ WriteBufferFromFileDescriptor message(STDERR_FILENO, 1024);
+
static size_t increment = 0;
static const char * indicators[8] =
{
@@ -1381,13 +1397,15 @@ private:
};
if (written_progress_chars)
- clearProgress();
+ message << RESTORE_CURSOR_POSITION CLEAR_TO_END_OF_LINE;
else
- std::cerr << SAVE_CURSOR_POSITION;
+ message << SAVE_CURSOR_POSITION;
+
+ message << DISABLE_LINE_WRAPPING;
+
+ size_t prefix_size = message.count();
- std::stringstream message;
message << indicators[increment % 8]
- << std::fixed << std::setprecision(3)
<< " Progress: ";
message
@@ -1402,8 +1420,7 @@ private:
else
message << ". ";
- written_progress_chars = message.str().size() - (increment % 8 == 7 ? 10 : 13);
- std::cerr << DISABLE_LINE_WRAPPING << message.rdbuf();
+ written_progress_chars = message.count() - prefix_size - (increment % 8 == 7 ? 10 : 13); /// Don't count invisible output (escape sequences).
/// If the approximate number of rows to process is known, we can display a progress bar and percentage.
if (progress.total_rows > 0)
@@ -1425,19 +1442,21 @@ private:
if (width_of_progress_bar > 0)
{
std::string bar = UnicodeBar::render(UnicodeBar::getWidth(progress.rows, 0, total_rows_corrected, width_of_progress_bar));
- std::cerr << "\033[0;32m" << bar << "\033[0m";
+ message << "\033[0;32m" << bar << "\033[0m";
if (width_of_progress_bar > static_cast(bar.size() / UNICODE_BAR_CHAR_SIZE))
- std::cerr << std::string(width_of_progress_bar - bar.size() / UNICODE_BAR_CHAR_SIZE, ' ');
+ message << std::string(width_of_progress_bar - bar.size() / UNICODE_BAR_CHAR_SIZE, ' ');
}
}
}
/// Underestimate percentage a bit to avoid displaying 100%.
- std::cerr << ' ' << (99 * progress.rows / total_rows_corrected) << '%';
+ message << ' ' << (99 * progress.rows / total_rows_corrected) << '%';
}
- std::cerr << ENABLE_LINE_WRAPPING;
+ message << ENABLE_LINE_WRAPPING;
++increment;
+
+ message.next();
}
@@ -1504,7 +1523,7 @@ private:
void showClientVersion()
{
- std::cout << DBMS_NAME << " client version " << VERSION_STRING << "." << std::endl;
+ std::cout << DBMS_NAME << " client version " << VERSION_STRING << VERSION_OFFICIAL << "." << std::endl;
}
public:
@@ -1569,7 +1588,7 @@ public:
}
}
- ioctl(0, TIOCGWINSZ, &terminal_size);
+ stdin_is_not_tty = !isatty(STDIN_FILENO);
namespace po = boost::program_options;
@@ -1577,7 +1596,11 @@ public:
unsigned min_description_length = line_length / 2;
if (!stdin_is_not_tty)
{
- line_length = std::max(3U, static_cast(terminal_size.ws_col));
+ if (ioctl(STDIN_FILENO, TIOCGWINSZ, &terminal_size))
+ throwFromErrno("Cannot obtain terminal window size (ioctl TIOCGWINSZ)", ErrorCodes::SYSTEM_ERROR);
+ line_length = std::max(
+ static_cast(strlen("--http_native_compression_disable_checksumming_on_decompress ")),
+ static_cast(terminal_size.ws_col));
min_description_length = std::min(min_description_length, line_length - 2);
}
diff --git a/dbms/programs/client/Suggest.h b/dbms/programs/client/Suggest.h
index 1be611d97b9..6120f875d57 100644
--- a/dbms/programs/client/Suggest.h
+++ b/dbms/programs/client/Suggest.h
@@ -39,7 +39,7 @@ private:
"DATABASES", "LIKE", "PROCESSLIST", "CASE", "WHEN", "THEN", "ELSE", "END", "DESCRIBE", "DESC", "USE", "SET", "OPTIMIZE", "FINAL", "DEDUPLICATE",
"INSERT", "VALUES", "SELECT", "DISTINCT", "SAMPLE", "ARRAY", "JOIN", "GLOBAL", "LOCAL", "ANY", "ALL", "INNER", "LEFT", "RIGHT", "FULL", "OUTER",
"CROSS", "USING", "PREWHERE", "WHERE", "GROUP", "BY", "WITH", "TOTALS", "HAVING", "ORDER", "COLLATE", "LIMIT", "UNION", "AND", "OR", "ASC", "IN",
- "KILL", "QUERY", "SYNC", "ASYNC", "TEST", "BETWEEN"
+ "KILL", "QUERY", "SYNC", "ASYNC", "TEST", "BETWEEN", "TRUNCATE"
};
/// Words are fetched asynchonously.
diff --git a/dbms/programs/compressor/Compressor.cpp b/dbms/programs/compressor/Compressor.cpp
index de51f16833e..427d58cbdc6 100644
--- a/dbms/programs/compressor/Compressor.cpp
+++ b/dbms/programs/compressor/Compressor.cpp
@@ -1,6 +1,7 @@
#include
#include
#include
+#include
#include
#include
@@ -9,6 +10,8 @@
#include
#include
#include
+#include
+#include
#include
@@ -64,7 +67,7 @@ int mainEntryClickHouseCompressor(int argc, char ** argv)
("hc", "use LZ4HC instead of LZ4")
("zstd", "use ZSTD instead of LZ4")
("codec", boost::program_options::value>()->multitoken(), "use codecs combination instead of LZ4")
- ("level", boost::program_options::value>()->multitoken(), "compression levels for codecs specified via --codec")
+ ("level", boost::program_options::value(), "compression level for codecs spicified via flags")
("none", "use no compression instead of LZ4")
("stat", "print block statistics of compressed data")
;
@@ -94,6 +97,9 @@ int mainEntryClickHouseCompressor(int argc, char ** argv)
if ((use_lz4hc || use_zstd || use_none) && !codecs.empty())
throw DB::Exception("Wrong options, codec flags like --zstd and --codec options are mutually exclusive", DB::ErrorCodes::BAD_ARGUMENTS);
+ if (!codecs.empty() && options.count("level"))
+ throw DB::Exception("Wrong options, --level is not compatible with --codec list", DB::ErrorCodes::BAD_ARGUMENTS);
+
std::string method_family = "LZ4";
if (use_lz4hc)
@@ -103,28 +109,22 @@ int mainEntryClickHouseCompressor(int argc, char ** argv)
else if (use_none)
method_family = "NONE";
- std::vector levels;
+ std::optional level = std::nullopt;
if (options.count("level"))
- levels = options["level"].as>();
+ level = options["level"].as();
+
DB::CompressionCodecPtr codec;
if (!codecs.empty())
{
- if (levels.size() > codecs.size())
- throw DB::Exception("Specified more levels than codecs", DB::ErrorCodes::BAD_ARGUMENTS);
+ DB::ParserCodec codec_parser;
- std::vector codec_names;
- for (size_t i = 0; i < codecs.size(); ++i)
- {
- if (i < levels.size())
- codec_names.emplace_back(codecs[i], levels[i]);
- else
- codec_names.emplace_back(codecs[i], std::nullopt);
- }
- codec = DB::CompressionCodecFactory::instance().get(codec_names);
+ std::string codecs_line = boost::algorithm::join(codecs, ",");
+ auto ast = DB::parseQuery(codec_parser, "(" + codecs_line + ")", 0);
+ codec = DB::CompressionCodecFactory::instance().get(ast, nullptr);
}
else
- codec = DB::CompressionCodecFactory::instance().get(method_family, levels.empty() ? std::nullopt : std::optional(levels.back()));
+ codec = DB::CompressionCodecFactory::instance().get(method_family, level);
DB::ReadBufferFromFileDescriptor rb(STDIN_FILENO);
diff --git a/dbms/programs/compressor/README.md b/dbms/programs/compressor/README.md
index 92dfe50cbc1..44a1f052824 100644
--- a/dbms/programs/compressor/README.md
+++ b/dbms/programs/compressor/README.md
@@ -17,11 +17,11 @@ $ ./clickhouse-compressor --decompress < input_file > output_file
Compress data with ZSTD at level 5:
```
-$ ./clickhouse-compressor --codec ZSTD --level 5 < input_file > output_file
+$ ./clickhouse-compressor --codec 'ZSTD(5)' < input_file > output_file
```
-Compress data with ZSTD level 10, LZ4HC level 7 and LZ4.
+Compress data with Delta of four bytes and ZSTD level 10.
```
-$ ./clickhouse-compressor --codec ZSTD --level 5 --codec LZ4HC --level 7 --codec LZ4 < input_file > output_file
+$ ./clickhouse-compressor --codec 'Delta(4)' --codec 'ZSTD(10)' < input_file > output_file
```
diff --git a/dbms/programs/copier/ClusterCopier.cpp b/dbms/programs/copier/ClusterCopier.cpp
index 588c9984f58..75096df74ed 100644
--- a/dbms/programs/copier/ClusterCopier.cpp
+++ b/dbms/programs/copier/ClusterCopier.cpp
@@ -1,7 +1,6 @@
#include "ClusterCopier.h"
#include
-
#include
#include
#include
@@ -13,14 +12,11 @@
#include
#include
#include
-
#include
#include
-
#include
#include
#include
-
#include
#include
#include
@@ -61,6 +57,7 @@
#include
#include
#include
+#include
#include
#include
#include
@@ -483,7 +480,7 @@ String DB::TaskShard::getHostNameExample() const
static bool isExtendedDefinitionStorage(const ASTPtr & storage_ast)
{
- const ASTStorage & storage = typeid_cast(*storage_ast);
+ const auto & storage = storage_ast->as();
return storage.partition_by || storage.order_by || storage.sample_by;
}
@@ -491,8 +488,8 @@ static ASTPtr extractPartitionKey(const ASTPtr & storage_ast)
{
String storage_str = queryToString(storage_ast);
- const ASTStorage & storage = typeid_cast(*storage_ast);
- const ASTFunction & engine = typeid_cast(*storage.engine);
+ const auto & storage = storage_ast->as();
+ const auto & engine = storage.engine->as