merged with master

.github/ISSUE_TEMPLATE/backward-compatibility.md

@@ -17,7 +17,7 @@ A clear and concise description of what works not as it is supposed to.
 * Which interface to use, if matters
 * Non-default settings, if any
 * `CREATE TABLE` statements for all tables involved
-* Sample data for all these tables, use [clickhouse-obfuscator](https://github.com/ClickHouse/ClickHouse/blob/master/dbms/programs/obfuscator/Obfuscator.cpp#L42-L80) if necessary
+* Sample data for all these tables, use [clickhouse-obfuscator](https://github.com/ClickHouse/ClickHouse/blob/master/programs/obfuscator/Obfuscator.cpp#L42-L80) if necessary
 * Queries to run that lead to unexpected result
 
 **Error message and/or stacktrace**

.github/ISSUE_TEMPLATE/bug-report.md

@@ -17,7 +17,7 @@ A clear and concise description of what works not as it is supposed to.
 * Which interface to use, if matters
 * Non-default settings, if any
 * `CREATE TABLE` statements for all tables involved
-* Sample data for all these tables, use [clickhouse-obfuscator](https://github.com/ClickHouse/ClickHouse/blob/master/dbms/programs/obfuscator/Obfuscator.cpp#L42-L80) if necessary
+* Sample data for all these tables, use [clickhouse-obfuscator](https://github.com/ClickHouse/ClickHouse/blob/master/programs/obfuscator/Obfuscator.cpp#L42-L80) if necessary
 * Queries to run that lead to unexpected result
 
 **Expected behavior**

.github/ISSUE_TEMPLATE/performance-issue.md

@@ -17,7 +17,7 @@ What exactly works slower than expected?
 * Which interface to use, if matters
 * Non-default settings, if any
 * `CREATE TABLE` statements for all tables involved
-* Sample data for all these tables, use [clickhouse-obfuscator](https://github.com/ClickHouse/ClickHouse/blob/master/dbms/programs/obfuscator/Obfuscator.cpp#L42-L80) if necessary
+* Sample data for all these tables, use [clickhouse-obfuscator](https://github.com/ClickHouse/ClickHouse/blob/master/programs/obfuscator/Obfuscator.cpp#L42-L80) if necessary
 * Queries to run that lead to slow performance
 
 **Expected performance**

.github/ISSUE_TEMPLATE/unexpected-behaviour.md

@@ -17,7 +17,7 @@ A clear and concise description of what works not as it is supposed to.
 * Which interface to use, if matters
 * Non-default settings, if any
 * `CREATE TABLE` statements for all tables involved
-* Sample data for all these tables, use [clickhouse-obfuscator](https://github.com/ClickHouse/ClickHouse/blob/master/dbms/programs/obfuscator/Obfuscator.cpp#L42-L80) if necessary
+* Sample data for all these tables, use [clickhouse-obfuscator](https://github.com/ClickHouse/ClickHouse/blob/master/programs/obfuscator/Obfuscator.cpp#L42-L80) if necessary
 * Queries to run that lead to unexpected result
 
 **Expected behavior**

.github/ISSUE_TEMPLATE/usability-issue.md

@@ -17,7 +17,7 @@ A clear and concise description of what works not as it is supposed to.
 * Which interface to use, if matters
 * Non-default settings, if any
 * `CREATE TABLE` statements for all tables involved
-* Sample data for all these tables, use [clickhouse-obfuscator](https://github.com/ClickHouse/ClickHouse/blob/master/dbms/programs/obfuscator/Obfuscator.cpp#L42-L80) if necessary
+* Sample data for all these tables, use [clickhouse-obfuscator](https://github.com/ClickHouse/ClickHouse/blob/master/programs/obfuscator/Obfuscator.cpp#L42-L80) if necessary
 * Queries to run that lead to unexpected result
 
 **Expected behavior**

.gitignore

@@ -73,100 +73,100 @@ contrib/libpoco/Poco/
 contrib/libpoco/bin/
 contrib/libpoco/cmake_uninstall.cmake
 contrib/libre2/re2_st/
-dbms/src/Client/clickhouse-benchmark
-dbms/src/Client/clickhouse-client
-dbms/src/Client/tests/test-connect
-dbms/src/Common/tests/arena_with_free_lists
-dbms/src/Common/tests/auto_array
-dbms/src/Common/tests/compact_array
-dbms/src/Common/tests/hash_table
-dbms/src/Common/tests/hashes_test
-dbms/src/Common/tests/int_hashes_perf
-dbms/src/Common/tests/lru_cache
-dbms/src/Common/tests/parallel_aggregation
-dbms/src/Common/tests/parallel_aggregation2
-dbms/src/Common/tests/radix_sort
-dbms/src/Common/tests/shell_command_test
-dbms/src/Common/tests/simple_cache
-dbms/src/Common/tests/sip_hash
-dbms/src/Common/tests/sip_hash_perf
-dbms/src/Common/tests/small_table
-dbms/src/Core/tests/exception
-dbms/src/Core/tests/field
-dbms/src/Core/tests/rvo_test
-dbms/src/Core/tests/string_pool
-dbms/src/DataStreams/tests/aggregating_stream
-dbms/src/DataStreams/tests/block_tab_separated_streams
-dbms/src/DataStreams/tests/collapsing_sorted_stream
-dbms/src/DataStreams/tests/expression_stream
-dbms/src/DataStreams/tests/filter_stream
-dbms/src/DataStreams/tests/filter_stream_hitlog
-dbms/src/DataStreams/tests/fork_streams
-dbms/src/DataStreams/tests/glue_streams
-dbms/src/DataStreams/tests/json_streams
-dbms/src/DataStreams/tests/native_streams
-dbms/src/DataStreams/tests/sorting_stream
-dbms/src/DataStreams/tests/tab_separated_streams
-dbms/src/DataStreams/tests/union_stream
-dbms/src/DataStreams/tests/union_stream2
-dbms/src/DataTypes/tests/data_type_string
-dbms/src/DataTypes/tests/data_types_number_fixed
-dbms/src/Functions/tests/functions_arithmetic
-dbms/src/Functions/tests/logical_functions_performance
-dbms/src/Functions/tests/number_traits
-dbms/src/IO/tests/async_write
-dbms/src/IO/tests/cached_compressed_read_buffer
-dbms/src/IO/tests/compressed_buffer
-dbms/src/IO/tests/hashing_read_buffer
-dbms/src/IO/tests/hashing_write_buffer
-dbms/src/IO/tests/io_and_exceptions
-dbms/src/IO/tests/io_operators
-dbms/src/IO/tests/mempbrk
-dbms/src/IO/tests/o_direct_and_dirty_pages
-dbms/src/IO/tests/parse_int_perf
-dbms/src/IO/tests/parse_int_perf2
-dbms/src/IO/tests/read_buffer
-dbms/src/IO/tests/read_buffer_aio
-dbms/src/IO/tests/read_buffer_perf
-dbms/src/IO/tests/read_escaped_string
-dbms/src/IO/tests/read_float_perf
-dbms/src/IO/tests/read_write_int
-dbms/src/IO/tests/valid_utf8
-dbms/src/IO/tests/valid_utf8_perf
-dbms/src/IO/tests/var_uint
-dbms/src/IO/tests/write_buffer
-dbms/src/IO/tests/write_buffer_aio
-dbms/src/IO/tests/write_buffer_perf
-dbms/src/Interpreters/tests/address_patterns
-dbms/src/Interpreters/tests/aggregate
-dbms/src/Interpreters/tests/compiler_test
-dbms/src/Interpreters/tests/create_query
-dbms/src/Interpreters/tests/expression
-dbms/src/Interpreters/tests/expression_analyzer
-dbms/src/Interpreters/tests/hash_map
-dbms/src/Interpreters/tests/hash_map2
-dbms/src/Interpreters/tests/hash_map3
-dbms/src/Interpreters/tests/hash_map_string
-dbms/src/Interpreters/tests/hash_map_string_2
-dbms/src/Interpreters/tests/hash_map_string_3
-dbms/src/Interpreters/tests/hash_map_string_small
-dbms/src/Interpreters/tests/in_join_subqueries_preprocessor
-dbms/src/Interpreters/tests/logical_expressions_optimizer
-dbms/src/Interpreters/tests/select_query
-dbms/src/Interpreters/tests/two_level_hash_map
-dbms/src/Interpreters/tests/users
-dbms/src/Parsers/tests/create_parser
-dbms/src/Parsers/tests/select_parser
-dbms/src/Server/clickhouse-server
-dbms/src/Server/clickhouse-server.init
-dbms/src/Storages/tests/hit_log
-dbms/src/Storages/tests/merge_tree
-dbms/src/Storages/tests/part_checker
-dbms/src/Storages/tests/part_name
-dbms/src/Storages/tests/pk_condition
-dbms/src/Storages/tests/seek_speed_test
-dbms/src/Storages/tests/storage_log
-dbms/src/Storages/tests/system_numbers
+dbms/Client/clickhouse-benchmark
+dbms/Client/clickhouse-client
+dbms/Client/tests/test-connect
+dbms/Common/tests/arena_with_free_lists
+dbms/Common/tests/auto_array
+dbms/Common/tests/compact_array
+dbms/Common/tests/hash_table
+dbms/Common/tests/hashes_test
+dbms/Common/tests/int_hashes_perf
+dbms/Common/tests/lru_cache
+dbms/Common/tests/parallel_aggregation
+dbms/Common/tests/parallel_aggregation2
+dbms/Common/tests/radix_sort
+dbms/Common/tests/shell_command_test
+dbms/Common/tests/simple_cache
+dbms/Common/tests/sip_hash
+dbms/Common/tests/sip_hash_perf
+dbms/Common/tests/small_table
+dbms/Core/tests/exception
+dbms/Core/tests/field
+dbms/Core/tests/rvo_test
+dbms/Core/tests/string_pool
+dbms/DataStreams/tests/aggregating_stream
+dbms/DataStreams/tests/block_tab_separated_streams
+dbms/DataStreams/tests/collapsing_sorted_stream
+dbms/DataStreams/tests/expression_stream
+dbms/DataStreams/tests/filter_stream
+dbms/DataStreams/tests/filter_stream_hitlog
+dbms/DataStreams/tests/fork_streams
+dbms/DataStreams/tests/glue_streams
+dbms/DataStreams/tests/json_streams
+dbms/DataStreams/tests/native_streams
+dbms/DataStreams/tests/sorting_stream
+dbms/DataStreams/tests/tab_separated_streams
+dbms/DataStreams/tests/union_stream
+dbms/DataStreams/tests/union_stream2
+dbms/DataTypes/tests/data_type_string
+dbms/DataTypes/tests/data_types_number_fixed
+dbms/Functions/tests/functions_arithmetic
+dbms/Functions/tests/logical_functions_performance
+dbms/Functions/tests/number_traits
+dbms/IO/tests/async_write
+dbms/IO/tests/cached_compressed_read_buffer
+dbms/IO/tests/compressed_buffer
+dbms/IO/tests/hashing_read_buffer
+dbms/IO/tests/hashing_write_buffer
+dbms/IO/tests/io_and_exceptions
+dbms/IO/tests/io_operators
+dbms/IO/tests/mempbrk
+dbms/IO/tests/o_direct_and_dirty_pages
+dbms/IO/tests/parse_int_perf
+dbms/IO/tests/parse_int_perf2
+dbms/IO/tests/read_buffer
+dbms/IO/tests/read_buffer_aio
+dbms/IO/tests/read_buffer_perf
+dbms/IO/tests/read_escaped_string
+dbms/IO/tests/read_float_perf
+dbms/IO/tests/read_write_int
+dbms/IO/tests/valid_utf8
+dbms/IO/tests/valid_utf8_perf
+dbms/IO/tests/var_uint
+dbms/IO/tests/write_buffer
+dbms/IO/tests/write_buffer_aio
+dbms/IO/tests/write_buffer_perf
+dbms/Interpreters/tests/address_patterns
+dbms/Interpreters/tests/aggregate
+dbms/Interpreters/tests/compiler_test
+dbms/Interpreters/tests/create_query
+dbms/Interpreters/tests/expression
+dbms/Interpreters/tests/expression_analyzer
+dbms/Interpreters/tests/hash_map
+dbms/Interpreters/tests/hash_map2
+dbms/Interpreters/tests/hash_map3
+dbms/Interpreters/tests/hash_map_string
+dbms/Interpreters/tests/hash_map_string_2
+dbms/Interpreters/tests/hash_map_string_3
+dbms/Interpreters/tests/hash_map_string_small
+dbms/Interpreters/tests/in_join_subqueries_preprocessor
+dbms/Interpreters/tests/logical_expressions_optimizer
+dbms/Interpreters/tests/select_query
+dbms/Interpreters/tests/two_level_hash_map
+dbms/Interpreters/tests/users
+dbms/Parsers/tests/create_parser
+dbms/Parsers/tests/select_parser
+dbms/Server/clickhouse-server
+dbms/Server/clickhouse-server.init
+dbms/Storages/tests/hit_log
+dbms/Storages/tests/merge_tree
+dbms/Storages/tests/part_checker
+dbms/Storages/tests/part_name
+dbms/Storages/tests/pk_condition
+dbms/Storages/tests/seek_speed_test
+dbms/Storages/tests/storage_log
+dbms/Storages/tests/system_numbers
 libs/libcommon/src/revision.h
 libs/libcommon/src/tests/date_lut2
 libs/libcommon/src/tests/date_lut3
@@ -184,15 +184,15 @@ libs/libzkutil/src/tests/zkutil_zookeeper_holder
 utils/zookeeper-create-entry-to-download-part/zookeeper-create-entry-to-download-part
 utils/zookeeper-dump-tree/zookeeper-dump-tree
 utils/zookeeper-remove-by-list/zookeeper-remove-by-list
-dbms/src/Storages/tests/remove_symlink_directory
+dbms/Storages/tests/remove_symlink_directory
 libs/libcommon/src/tests/json_test
 utils/compressor/zstd_test
 utils/wikistat-loader/wikistat-loader
-dbms/src/Common/tests/pod_array
+dbms/Common/tests/pod_array
 
-dbms/src/Server/data/*
-dbms/src/Server/metadata/*
-dbms/src/Server/status
+dbms/Server/data/*
+dbms/Server/metadata/*
+dbms/Server/status
 config-9001.xml
 
 *-preprocessed.xml
@@ -242,7 +242,7 @@ website/package-lock.json
 */.DS_Store
 
 # Ignore files for locally disabled tests
-/dbms/tests/queries/**/*.disabled
+/dbms/queries/**/*.disabled
 
 # cquery cache
 /.cquery-cache

.gitlab-ci.yml

@@ -31,11 +31,11 @@ build:
     - docker pull $CI_REGISTRY/yandex/clickhouse-builder
     - docker run --rm --volumes-from "${HOSTNAME}-build" --workdir "${CI_PROJECT_DIR}" --env CI_PROJECT_DIR=${CI_PROJECT_DIR} $CI_REGISTRY/yandex/clickhouse-builder /build_gitlab_ci.sh
   # You can upload your binary to nexus
-    - curl -v --keepalive-time 60 --keepalive --user "$NEXUS_USER:$NEXUS_PASSWORD" -XPUT "http://$NEXUS_HOST/repository/binaries/$CI_PROJECT_NAME" --upload-file ./dbms/src/Server/clickhouse
+    - curl -v --keepalive-time 60 --keepalive --user "$NEXUS_USER:$NEXUS_PASSWORD" -XPUT "http://$NEXUS_HOST/repository/binaries/$CI_PROJECT_NAME" --upload-file ./dbms/Server/clickhouse
   # Or download artifacts from gitlab
   artifacts:
     paths:
-      - ./dbms/src/Server/clickhouse
+      - ./dbms/Server/clickhouse
     expire_in: 1 day
   tags:
-    - docker
+    - docker

CHANGELOG.md

@@ -234,7 +234,7 @@
 * Updated checking for hung queries in clickhouse-test script [#8858](https://github.com/ClickHouse/ClickHouse/pull/8858) ([Alexander Kazakov](https://github.com/Akazz))
 * Removed some useless files from repository. [#8843](https://github.com/ClickHouse/ClickHouse/pull/8843) ([alexey-milovidov](https://github.com/alexey-milovidov))
 * Changed type of math perftests from `once` to `loop`. [#8783](https://github.com/ClickHouse/ClickHouse/pull/8783) ([Nikolai Kochetov](https://github.com/KochetovNicolai))
-* Add docker image which allows to build interactive code browser HTML report for our codebase. [#8781](https://github.com/ClickHouse/ClickHouse/pull/8781) ([alesapin](https://github.com/alesapin)) See [Woboq Code Browser](https://clickhouse-test-reports.s3.yandex.net/codebrowser/html_report///ClickHouse/dbms/src/index.html)
+* Add docker image which allows to build interactive code browser HTML report for our codebase. [#8781](https://github.com/ClickHouse/ClickHouse/pull/8781) ([alesapin](https://github.com/alesapin)) See [Woboq Code Browser](https://clickhouse-test-reports.s3.yandex.net/codebrowser/html_report///ClickHouse/dbms/index.html)
 * Suppress some test failures under MSan. [#8780](https://github.com/ClickHouse/ClickHouse/pull/8780) ([Alexander Kuzmenkov](https://github.com/akuzm))
 * Speedup "exception while insert" test. This test often time out in debug-with-coverage build. [#8711](https://github.com/ClickHouse/ClickHouse/pull/8711) ([alexey-milovidov](https://github.com/alexey-milovidov))
 * Updated `libcxx` and `libcxxabi` to master. In preparation to [#9304](https://github.com/ClickHouse/ClickHouse/issues/9304) [#9308](https://github.com/ClickHouse/ClickHouse/pull/9308) ([alexey-milovidov](https://github.com/alexey-milovidov))
@@ -341,7 +341,7 @@
 [#9513](https://github.com/ClickHouse/ClickHouse/pull/9513) [(filimonov)](https://github.com/filimonov)
 
 #### New Feature
-* Add `deduplicate_blocks_in_dependent_materialized_views` option to control the behaviour of idempotent inserts into tables with materialized views. This new feature was added to the bugfix release by a special request from Altinity. 
+* Add `deduplicate_blocks_in_dependent_materialized_views` option to control the behaviour of idempotent inserts into tables with materialized views. This new feature was added to the bugfix release by a special request from Altinity.
 [#9070](https://github.com/ClickHouse/ClickHouse/pull/9070) [(urykhy)](https://github.com/urykhy)
 
 ### ClickHouse release v20.1.2.4, 2020-01-22

CMakeLists.txt

@@ -353,17 +353,11 @@ if (ENABLE_TESTS)
 endif ()
 
 # Need to process before "contrib" dir:
-include (cmake/find/jemalloc.cmake)
 include (cmake/find/mysqlclient.cmake)
 
 # When testing for memory leaks with Valgrind, don't link tcmalloc or jemalloc.
 
-if (USE_JEMALLOC)
-    message (STATUS "Link jemalloc: ${JEMALLOC_LIBRARIES}")
-    set (MALLOC_LIBRARIES ${JEMALLOC_LIBRARIES})
-elseif (SANITIZE)
-    message (STATUS "Will use ${SANITIZE} sanitizer.")
-elseif (OS_LINUX)
+if (OS_LINUX AND NOT ENABLE_JEMALLOC)
     message (WARNING "Non default allocator is disabled. This is not recommended for production Linux builds.")
 endif ()
 
@@ -386,8 +380,13 @@ macro (add_executable target)
     endif()
 endmacro()
 
+set(ConfigIncludePath ${CMAKE_CURRENT_BINARY_DIR}/includes/configs CACHE INTERNAL "Path to generated configuration files.")
+include_directories(${ConfigIncludePath})
+
 add_subdirectory (base)
-add_subdirectory (utils)
 add_subdirectory (dbms)
+add_subdirectory (programs)
+add_subdirectory (tests)
+add_subdirectory (utils)
 
 include (cmake/print_include_directories.cmake)

README.md

@@ -8,14 +8,15 @@ ClickHouse is an open-source column-oriented database management system that all
 * [Tutorial](https://clickhouse.tech/docs/en/getting_started/tutorial/) shows how to set up and query small ClickHouse cluster.
 * [Documentation](https://clickhouse.tech/docs/en/) provides more in-depth information.
 * [YouTube channel](https://www.youtube.com/c/ClickHouseDB) has a lot of content about ClickHouse in video format.
-* [Slack](https://join.slack.com/t/clickhousedb/shared_invite/enQtOTUzMjM4ODQwNTc5LWJmMjE3Yjc2YmI1ZDBlZmI4ZTc3OWY3ZTIwYTljYzY4MzBlODM3YzBjZTc1YmYyODRlZTJkYTgzYzBiNTA2Yjk) and [Telegram](https://telegram.me/clickhouse_en) allow to chat with ClickHouse users in real-time.
+* [Slack](https://join.slack.com/t/clickhousedb/shared_invite/zt-d2zxkf9e-XyxDa_ucfPxzuH4SJIm~Ng) and [Telegram](https://telegram.me/clickhouse_en) allow to chat with ClickHouse users in real-time.
 * [Blog](https://clickhouse.yandex/blog/en/) contains various ClickHouse-related articles, as well as announces and reports about events.
 * [Contacts](https://clickhouse.tech/#contacts) can help to get your questions answered if there are any.
 * You can also [fill this form](https://forms.yandex.com/surveys/meet-yandex-clickhouse-team/) to meet Yandex ClickHouse team in person.
 
 ## Upcoming Events
 
+* [ClickHouse Online Meetup (in Russian)](https://events.yandex.ru/events/click-house-onlajn-vs-03-04-2020) on April 3, 2020.
+* [ClickHouse in Avito (online in Russian)](https://avitotech.timepad.ru/event/1290051/) on April 9, 2020.
 * [ClickHouse Workshop in Novosibirsk](https://2020.codefest.ru/lecture/1628) on TBD date.
 * [Talks on Saint HighLoad++ in St. Petersburg](https://www.highload.ru/spb/2020/abstracts/6647) on TBD date.
 * [Yandex C++ Open-Source Sprints in Moscow](https://events.yandex.ru/events/otkrytyj-kod-v-yandek-28-03-2020) on TBD date.
-* [ClickHouse in Avito (online in Russian)](https://avitotech.timepad.ru/event/1290051/) on April 9, 2020.

base/common/config_common.h.in

@@ -2,5 +2,4 @@
 
 // .h autogenerated by cmake !
 
-#cmakedefine01 USE_JEMALLOC
 #cmakedefine01 UNBUNDLED

cmake/Modules/FindJeMalloc.cmake

@@ -1,46 +0,0 @@
-# https://github.com/bro/cmake/blob/master/FindJeMalloc.cmake
-#
-# - Try to find jemalloc headers and libraries.
-#
-# Usage of this module as follows:
-#
-#     find_package(JeMalloc)
-#
-# Variables used by this module, they can change the default behaviour and need
-# to be set before calling find_package:
-#
-#  JEMALLOC_ROOT_DIR Set this variable to the root installation of
-#                    jemalloc if the module has problems finding
-#                    the proper installation path.
-#
-# Variables defined by this module:
-#
-#  JEMALLOC_FOUND             System has jemalloc libs/headers
-#  JEMALLOC_LIBRARIES         The jemalloc library/libraries
-#  JEMALLOC_INCLUDE_DIR       The location of jemalloc headers
-
-find_path(JEMALLOC_ROOT_DIR
-    NAMES include/jemalloc/jemalloc.h
-)
-
-find_library(JEMALLOC_LIBRARIES
-    NAMES jemalloc
-    HINTS ${JEMALLOC_ROOT_DIR}/lib
-)
-
-find_path(JEMALLOC_INCLUDE_DIR
-    NAMES jemalloc/jemalloc.h
-    HINTS ${JEMALLOC_ROOT_DIR}/include
-)
-
-include(FindPackageHandleStandardArgs)
-find_package_handle_standard_args(JeMalloc DEFAULT_MSG
-    JEMALLOC_LIBRARIES
-    JEMALLOC_INCLUDE_DIR
-)
-
-mark_as_advanced(
-    JEMALLOC_ROOT_DIR
-    JEMALLOC_LIBRARIES
-    JEMALLOC_INCLUDE_DIR
-)

cmake/find/jemalloc.cmake

@@ -1,42 +0,0 @@
-if (OS_LINUX AND NOT SANITIZE AND NOT ARCH_32 AND NOT ARCH_PPC64LE)
-    set(ENABLE_JEMALLOC_DEFAULT ${ENABLE_LIBRARIES})
-else ()
-    set(ENABLE_JEMALLOC_DEFAULT 0)
-endif ()
-
-option (ENABLE_JEMALLOC "Set to TRUE to use jemalloc" ${ENABLE_JEMALLOC_DEFAULT})
-if (OS_LINUX)
-    option (USE_INTERNAL_JEMALLOC_LIBRARY "Set to FALSE to use system jemalloc library instead of bundled" ${NOT_UNBUNDLED})
-else()
-    option (USE_INTERNAL_JEMALLOC_LIBRARY "Set to FALSE to use system jemalloc library instead of bundled" OFF)
-endif()
-
-if (ENABLE_JEMALLOC)
-    if (USE_INTERNAL_JEMALLOC_LIBRARY AND NOT EXISTS "${ClickHouse_SOURCE_DIR}/contrib/jemalloc/src/jemalloc.c")
-       message (WARNING "submodule contrib/jemalloc is missing. to fix try run: \n git submodule update --init --recursive")
-       set (USE_INTERNAL_JEMALLOC_LIBRARY 0)
-       set (MISSING_INTERNAL_JEMALLOC_LIBRARY 1)
-    endif ()
-
-    if (NOT USE_INTERNAL_JEMALLOC_LIBRARY)
-        find_package (JeMalloc)
-    endif ()
-
-    if ((NOT JEMALLOC_LIBRARIES OR NOT JEMALLOC_INCLUDE_DIR) AND NOT MISSING_INTERNAL_JEMALLOC_LIBRARY )
-        set (JEMALLOC_LIBRARIES "jemalloc")
-        set (JEMALLOC_INCLUDE_DIR "${ClickHouse_SOURCE_DIR}/contrib/jemalloc-cmake/include" "${ClickHouse_SOURCE_DIR}/contrib/jemalloc-cmake/include_linux_x86_64")
-        set (USE_INTERNAL_JEMALLOC_LIBRARY 1)
-    endif ()
-
-    if (JEMALLOC_LIBRARIES)
-        set (USE_JEMALLOC 1)
-    elseif (NOT MISSING_INTERNAL_JEMALLOC_LIBRARY)
-        message (FATAL_ERROR "ENABLE_JEMALLOC is set to true, but library was not found")
-    endif ()
-
-    if (SANITIZE)
-        message (FATAL_ERROR "ENABLE_JEMALLOC is set to true, but it cannot be used with sanitizers")
-    endif ()
-
-    message (STATUS "Using jemalloc=${USE_JEMALLOC}: ${JEMALLOC_INCLUDE_DIR} : ${JEMALLOC_LIBRARIES}")
-endif ()

cmake/lib_name.cmake

@@ -1,5 +1,5 @@
 set(DIVIDE_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/contrib/libdivide)
-set(DBMS_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/dbms/src ${ClickHouse_BINARY_DIR}/dbms/src)
+set(DBMS_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/dbms ${ClickHouse_BINARY_DIR}/dbms)
 set(DOUBLE_CONVERSION_CONTRIB_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/contrib/double-conversion)
 set(METROHASH_CONTRIB_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/contrib/libmetrohash/src)
 set(PCG_RANDOM_INCLUDE_DIR ${ClickHouse_SOURCE_DIR}/contrib/libpcg-random/include)

cmake/sanitize.cmake

@@ -23,7 +23,7 @@ if (SANITIZE)
         # RelWithDebInfo, and downgrade optimizations to -O1 but not to -Og, to
         # keep the binary size down.
         # TODO: try compiling with -Og and with ld.gold.
-        set (MSAN_FLAGS "-fsanitize=memory -fsanitize-memory-track-origins -fno-optimize-sibling-calls -fsanitize-blacklist=${CMAKE_SOURCE_DIR}/dbms/tests/msan_suppressions.txt")
+        set (MSAN_FLAGS "-fsanitize=memory -fsanitize-memory-track-origins -fno-optimize-sibling-calls -fsanitize-blacklist=${CMAKE_SOURCE_DIR}/tests/msan_suppressions.txt")
 
         set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${SAN_FLAGS} ${MSAN_FLAGS}")
         set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} ${SAN_FLAGS} ${MSAN_FLAGS}")

contrib/CMakeLists.txt

@@ -77,9 +77,7 @@ endif ()
 
 add_subdirectory (cctz-cmake)
 
-if (ENABLE_JEMALLOC AND USE_INTERNAL_JEMALLOC_LIBRARY)
-    add_subdirectory (jemalloc-cmake)
-endif ()
+add_subdirectory (jemalloc-cmake)
 
 if (USE_INTERNAL_CPUID_LIBRARY)
     add_subdirectory (libcpuid)

contrib/jemalloc-cmake/CMakeLists.txt

@@ -1,70 +1,110 @@
-set(JEMALLOC_SOURCE_DIR ${ClickHouse_SOURCE_DIR}/contrib/jemalloc)
+option (ENABLE_JEMALLOC "Enable jemalloc allocator" ${ENABLE_LIBRARIES})
 
-set(SRCS
-${JEMALLOC_SOURCE_DIR}/src/arena.c
-${JEMALLOC_SOURCE_DIR}/src/background_thread.c
-${JEMALLOC_SOURCE_DIR}/src/base.c
-${JEMALLOC_SOURCE_DIR}/src/bin.c
-${JEMALLOC_SOURCE_DIR}/src/bitmap.c
-${JEMALLOC_SOURCE_DIR}/src/ckh.c
-${JEMALLOC_SOURCE_DIR}/src/ctl.c
-${JEMALLOC_SOURCE_DIR}/src/div.c
-${JEMALLOC_SOURCE_DIR}/src/extent.c
-${JEMALLOC_SOURCE_DIR}/src/extent_dss.c
-${JEMALLOC_SOURCE_DIR}/src/extent_mmap.c
-${JEMALLOC_SOURCE_DIR}/src/hash.c
-${JEMALLOC_SOURCE_DIR}/src/hook.c
-${JEMALLOC_SOURCE_DIR}/src/jemalloc.c
-${JEMALLOC_SOURCE_DIR}/src/large.c
-${JEMALLOC_SOURCE_DIR}/src/log.c
-${JEMALLOC_SOURCE_DIR}/src/malloc_io.c
-${JEMALLOC_SOURCE_DIR}/src/mutex.c
-${JEMALLOC_SOURCE_DIR}/src/mutex_pool.c
-${JEMALLOC_SOURCE_DIR}/src/nstime.c
-${JEMALLOC_SOURCE_DIR}/src/pages.c
-${JEMALLOC_SOURCE_DIR}/src/prng.c
-${JEMALLOC_SOURCE_DIR}/src/prof.c
-${JEMALLOC_SOURCE_DIR}/src/rtree.c
-${JEMALLOC_SOURCE_DIR}/src/sc.c
-${JEMALLOC_SOURCE_DIR}/src/stats.c
-${JEMALLOC_SOURCE_DIR}/src/sz.c
-${JEMALLOC_SOURCE_DIR}/src/tcache.c
-${JEMALLOC_SOURCE_DIR}/src/test_hooks.c
-${JEMALLOC_SOURCE_DIR}/src/ticker.c
-${JEMALLOC_SOURCE_DIR}/src/tsd.c
-${JEMALLOC_SOURCE_DIR}/src/witness.c
-)
-
-if(OS_DARWIN)
-  list(APPEND SRCS ${JEMALLOC_SOURCE_DIR}/src/zone.c)
-endif()
-
-if (CMAKE_CXX_COMPILER_ID STREQUAL "GNU")
-    set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -w")
+if (SANITIZE OR NOT OS_LINUX OR NOT (ARCH_AMD64 OR ARCH_ARM))
+    set (ENABLE_JEMALLOC OFF)
+    message (STATUS "jemalloc is disabled implicitly: it doesn't work with sanitizers and can only be used on Linux with x86_64 or aarch64.")
 endif ()
 
-add_library(jemalloc STATIC ${SRCS})
+if (ENABLE_JEMALLOC)
+    option (USE_INTERNAL_JEMALLOC "Use internal jemalloc library" ${NOT_UNBUNDLED})
 
-target_include_directories(jemalloc PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/include)
+    if (USE_INTERNAL_JEMALLOC)
+        set (LIBRARY_DIR "${ClickHouse_SOURCE_DIR}/contrib/jemalloc")
 
-if (ARCH_AMD64)
-    target_include_directories(jemalloc PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/include_linux_x86_64)
-elseif (ARCH_ARM)
-    target_include_directories(jemalloc PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/include_linux_aarch64)
-else ()
-    message (FATAL_ERROR "jemalloc can only be used on x86_64 or aarch64.")
-endif ()
+        set (SRCS
+            ${LIBRARY_DIR}/src/arena.c
+            ${LIBRARY_DIR}/src/background_thread.c
+            ${LIBRARY_DIR}/src/base.c
+            ${LIBRARY_DIR}/src/bin.c
+            ${LIBRARY_DIR}/src/bitmap.c
+            ${LIBRARY_DIR}/src/ckh.c
+            ${LIBRARY_DIR}/src/ctl.c
+            ${LIBRARY_DIR}/src/div.c
+            ${LIBRARY_DIR}/src/extent.c
+            ${LIBRARY_DIR}/src/extent_dss.c
+            ${LIBRARY_DIR}/src/extent_mmap.c
+            ${LIBRARY_DIR}/src/hash.c
+            ${LIBRARY_DIR}/src/hook.c
+            ${LIBRARY_DIR}/src/jemalloc.c
+            ${LIBRARY_DIR}/src/large.c
+            ${LIBRARY_DIR}/src/log.c
+            ${LIBRARY_DIR}/src/malloc_io.c
+            ${LIBRARY_DIR}/src/mutex.c
+            ${LIBRARY_DIR}/src/mutex_pool.c
+            ${LIBRARY_DIR}/src/nstime.c
+            ${LIBRARY_DIR}/src/pages.c
+            ${LIBRARY_DIR}/src/prng.c
+            ${LIBRARY_DIR}/src/prof.c
+            ${LIBRARY_DIR}/src/rtree.c
+            ${LIBRARY_DIR}/src/sc.c
+            ${LIBRARY_DIR}/src/stats.c
+            ${LIBRARY_DIR}/src/sz.c
+            ${LIBRARY_DIR}/src/tcache.c
+            ${LIBRARY_DIR}/src/test_hooks.c
+            ${LIBRARY_DIR}/src/ticker.c
+            ${LIBRARY_DIR}/src/tsd.c
+            ${LIBRARY_DIR}/src/witness.c
+        )
+        if (OS_DARWIN)
+            list(APPEND SRCS ${LIBRARY_DIR}/src/zone.c)
+        endif ()
 
-target_include_directories(jemalloc PRIVATE
-    ${JEMALLOC_SOURCE_DIR}/include)
+        add_library(jemalloc ${SRCS})
+        target_include_directories(jemalloc PRIVATE ${LIBRARY_DIR}/include)
+        target_include_directories(jemalloc PUBLIC include)
+        if (ARCH_AMD64)
+            target_include_directories(jemalloc PUBLIC include_linux_x86_64)
+        elseif (ARCH_ARM)
+            target_include_directories(jemalloc PUBLIC include_linux_aarch64)
+        endif ()
 
-target_compile_definitions(jemalloc PRIVATE -DJEMALLOC_NO_PRIVATE_NAMESPACE)
+        target_compile_definitions(jemalloc PRIVATE -DJEMALLOC_NO_PRIVATE_NAMESPACE)
 
-if (CMAKE_BUILD_TYPE_UC STREQUAL "DEBUG")
-    target_compile_definitions(jemalloc PRIVATE -DJEMALLOC_DEBUG=1 -DJEMALLOC_PROF=1)
+        if (CMAKE_BUILD_TYPE_UC STREQUAL "DEBUG")
+            target_compile_definitions(jemalloc PRIVATE -DJEMALLOC_DEBUG=1 -DJEMALLOC_PROF=1)
 
-    if (USE_UNWIND)
-        target_compile_definitions (jemalloc PRIVATE -DJEMALLOC_PROF_LIBUNWIND=1)
-        target_link_libraries (jemalloc PRIVATE ${UNWIND_LIBRARIES})
+            if (USE_UNWIND)
+                target_compile_definitions (jemalloc PRIVATE -DJEMALLOC_PROF_LIBUNWIND=1)
+                target_link_libraries (jemalloc PRIVATE unwind)
+            endif ()
+        endif ()
+
+        target_compile_options(jemalloc PUBLIC -Wno-redundant-decls)
+    else ()
+        find_library(LIBRARY_JEMALLOC jemalloc)
+        find_path(INCLUDE_JEMALLOC jemalloc/jemalloc.h)
+
+        set(THREADS_PREFER_PTHREAD_FLAG ON)
+        find_package(Threads REQUIRED)
+
+        add_library (jemalloc STATIC IMPORTED)
+        set_property (TARGET jemalloc PROPERTY IMPORTED_LOCATION ${LIBRARY_JEMALLOC})
+        set_property (TARGET jemalloc PROPERTY INTERFACE_INCLUDE_DIRECTORIES ${INCLUDE_JEMALLOC})
+        set_property (TARGET jemalloc PROPERTY INTERFACE_LINK_LIBRARIES Threads::Threads dl)
+
+        set (CMAKE_REQUIRED_LIBRARIES jemalloc)
+        check_cxx_source_compiles (
+            "
+            #include <jemalloc/jemalloc.h>
+
+            int main() {
+                free(mallocx(1, 0));
+            }
+            "
+            EXTERNAL_JEMALLOC_WORKS
+        )
+
+        if (NOT EXTERNAL_JEMALLOC_WORKS)
+            message (FATAL_ERROR "jemalloc is unusable: ${LIBRARY_JEMALLOC} ${INCLUDE_JEMALLOC}")
+        endif ()
     endif ()
+
+    set_property(TARGET jemalloc APPEND PROPERTY INTERFACE_COMPILE_DEFINITIONS USE_JEMALLOC=1)
+
+    message (STATUS "Using jemalloc")
+else ()
+    add_library(jemalloc INTERFACE)
+    target_compile_definitions(jemalloc INTERFACE USE_JEMALLOC=0)
+
+    message (STATUS "Not using jemalloc")
 endif ()

dbms/Access/ContextAccess.cpp

@@ -408,8 +408,10 @@ boost::shared_ptr<const AccessRights> ContextAccess::calculateResultAccess(bool
     static const AccessFlags table_and_dictionary_ddl = table_ddl | dictionary_ddl;
     static const AccessFlags write_table_access = AccessType::INSERT | AccessType::OPTIMIZE;
     static const AccessFlags all_dcl = AccessType::CREATE_USER | AccessType::CREATE_ROLE | AccessType::CREATE_POLICY
-        | AccessType::CREATE_QUOTA | AccessType::ALTER_USER | AccessType::ALTER_POLICY | AccessType::ALTER_QUOTA | AccessType::DROP_USER
-        | AccessType::DROP_ROLE | AccessType::DROP_POLICY | AccessType::DROP_QUOTA | AccessType::ROLE_ADMIN;
+        | AccessType::CREATE_QUOTA | AccessType::CREATE_SETTINGS_PROFILE | AccessType::ALTER_USER | AccessType::ALTER_ROLE
+        | AccessType::ALTER_POLICY | AccessType::ALTER_QUOTA | AccessType::ALTER_SETTINGS_PROFILE | AccessType::DROP_USER
+        | AccessType::DROP_ROLE | AccessType::DROP_POLICY | AccessType::DROP_QUOTA | AccessType::DROP_SETTINGS_PROFILE
+        | AccessType::ROLE_ADMIN;
 
     if (readonly_)
         merged_access->revoke(write_table_access | all_dcl | table_and_dictionary_ddl | AccessType::SYSTEM | AccessType::KILL_QUERY);
@@ -540,14 +542,14 @@ std::shared_ptr<const ContextAccess> ContextAccess::getFullAccess()
 std::shared_ptr<const Settings> ContextAccess::getDefaultSettings() const
 {
     std::lock_guard lock{mutex};
-    return enabled_settings->getSettings();
+    return enabled_settings ? enabled_settings->getSettings() : nullptr;
 }
 
 
 std::shared_ptr<const SettingsConstraints> ContextAccess::getSettingsConstraints() const
 {
     std::lock_guard lock{mutex};
-    return enabled_settings->getConstraints();
+    return enabled_settings ? enabled_settings->getConstraints() : nullptr;
 }
 
 }