From 47798f0b8a8e67f81750a326dfbb941d62e5b495 Mon Sep 17 00:00:00 2001 From: proller Date: Tue, 10 Oct 2017 16:32:31 +0300 Subject: [PATCH 01/14] Fix PerformanceTest runs --- dbms/src/Server/PerformanceTest.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dbms/src/Server/PerformanceTest.cpp b/dbms/src/Server/PerformanceTest.cpp index 26e779bfac4..c975898eb37 100644 --- a/dbms/src/Server/PerformanceTest.cpp +++ b/dbms/src/Server/PerformanceTest.cpp @@ -931,7 +931,7 @@ private: else throw DB::Exception("Unknown type " + config_exec_type + " in :" + test_name); - times_to_run = test_config->getUInt("times_to_run"); + times_to_run = test_config->getUInt("times_to_run", 1); stop_conditions_by_run.clear(); TestStopConditions stop_conditions_template; From 16cd8fbda6e4a9346f48e7badaa9048362a8d324 Mon Sep 17 00:00:00 2001 From: proller Date: Tue, 10 Oct 2017 20:55:35 +0300 Subject: [PATCH 02/14] clickhouse_report.sh : more info about OS and kernel --- utils/report/clickhouse_report.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/utils/report/clickhouse_report.sh b/utils/report/clickhouse_report.sh index 369529b50e0..253b8812066 100755 --- a/utils/report/clickhouse_report.sh +++ b/utils/report/clickhouse_report.sh @@ -18,3 +18,5 @@ top -bn1 tail -n200 /var/log/clickhouse-server/clickhouse-server.err.log tail -n200 /var/log/clickhouse-server/clickhouse-server.log tail -n100 /var/log/clickhouse-server/stderr +cat /etc/lsb-release +uname -a From 5505013ba9837916d5da7a2c6d45ee1b48ad4cbe Mon Sep 17 00:00:00 2001 From: Ivan Blinkov Date: Wed, 11 Oct 2017 14:19:17 +0300 Subject: [PATCH 03/14] Switch links to next event (#1332) * update presentations * CLICKHOUSE-2936: redirect from clickhouse.yandex.ru and clickhouse.yandex.com * update submodule * lost files * CLICKHOUSE-2981: prefer sphinx docs over original reference * CLICKHOUSE-2981: docs styles more similar to main website + add flags to switch language links * update presentations * Less confusing directory structure (docs -> doc/reference/) * Minify sphinx docs too * Website release script: fail fast + pass docker hash on deploy * Do not underline links in docs * shorter * cleanup docker images * tune nginx config * CLICKHOUSE-3043: get rid of habrastorage links * Lost translation * CLICKHOUSE-2936: temporary client-side redirect * behaves weird in test * put redirect back * CLICKHOUSE-3047: copy docs txts to public too * move to proper file * remove old pages to avoid confusion * Remove reference redirect warning for now * Refresh README.md * Yellow buttons in docs * Use svg flags instead of unicode ones in docs * fix test website instance * Put flags to separate files * wrong flag * Copy Yandex.Metrica introduction from main page to docs * Yet another home page structure change, couple new blocks (CLICKHOUSE-3045) * Update Contacts section * CLICKHOUSE-2849: more detailed legal information * CLICKHOUSE-2978 preparation - split by files * More changes in Contacts block * Tune texts on index page * update presentations * One more benchmark * Add usage sections to index page, adapted from slides * Get the roadmap started, based on slides from last ClickHouse Meetup * CLICKHOUSE-2977: some rendering tuning * Get rid of excessive section in the end of getting started * Make headers linkable * CLICKHOUSE-2981: links to editing reference - https://github.com/yandex/ClickHouse/issues/849 * CLICKHOUSE-2981: fix mobile styles in docs * Ban crawling of duplicating docs * Open some external links in new tab * Ban old docs too * Lots of trivial fixes in english docs * Lots of trivial fixes in russian docs * Remove getting started copies in markdown * Add Yandex.Webmaster * Fix some sphinx warnings * More warnings fixed in english docs * More sphinx warnings fixed * Add code-block:: text * More code-block:: text * These headers look not that well * Better switch between documentation languages * merge use_case.rst into ya_metrika_task.rst * Edit the agg_functions.rst texts * Add lost empty lines * Lost blank lines * Add new logo sizes * update presentations * Next step in migrating to new documentation * Fix all warnings in en reference * Fix all warnings in ru reference * Re-arrange existing reference * Move operation tips to main reference * Fix typos noticed by milovidov@ * Get rid of zookeeper.md * Looks like duplicate of tutorial.html * Fix some mess with html tags in tutorial * No idea why nobody noticed this before, but it was completely not clear whet to get the data * Match code block styling between main and tutorial pages (in favor of the latter) * Get rid of some copypaste in tutorial * Normalize header styles * Move example_datasets to sphinx * Move presentations submodule to website * Move and update README.md * No point in duplicating articles from habrahabr here * Move development-related docs as is for now * doc/reference/ -> docs/ (to match the URL on website) * Adapt links to match the previous commit * Adapt development docs to rst (still lacks translation and strikethrough support) * clean on release * blacklist presentations in gulp * strikethrough support in sphinx * just copy development folder for now * fix weird introduction in style article * Style guide translation (WIP) * Finish style guide translation to English * gulp clean separately * Update year in LICENSE * Initial CONTRIBUTING.md * Fix remaining links to old docs in tutorial * Some tutorial fixes * Typo * Another typo * Update list of authors from yandex-team accoding to git log * Fix diff with master * couple fixes in en what_is_clickhouse.rst * Try different link to blog in Russian * Swap words * Slightly larger line height * CLICKHOUSE-3089: disable hyphenation in docs * update presentations * Fix copying of txt files * update submodule * CLICKHOUSE-3108: fix overflow issues in mobile version * Less weird tutorial header in mobile version * CLICKHOUSE-3073: skip sourcemaps by default * CLICKHOUSE-3067: rename item in docs navigation * fix list markup * CLICKHOUSE-3067: some documentation style tuning * CLICKHOUSE-3067: less laggy single page documentation * update presentations * YQL-3278: add some links to ClickHouse Meetup in Berlin on October 5, 2017 * Add "time series" keyword * Switch link to next event * Switch link to next event #2 * smaller font --- README.md | 2 +- docs/en/conf.py | 1 - website/index.css | 2 +- website/index.html | 2 +- 4 files changed, 3 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 9ac4e4cc9d2..b48aed5ac0a 100644 --- a/README.md +++ b/README.md @@ -2,4 +2,4 @@ ClickHouse is an open-source column-oriented database management system that all [Read more...](https://clickhouse.yandex/) -[ClickHouse Meetup in Berlin on October 5, 2017](https://events.yandex.com/events/meetings/05-10-2017/) +[ClickHouse Community Meetup in Palo Alto on October 25, 2017](http://bit.ly/clickhouse-meetup-palo-alto-october-2017) diff --git a/docs/en/conf.py b/docs/en/conf.py index b35129346ce..c58139ce779 100644 --- a/docs/en/conf.py +++ b/docs/en/conf.py @@ -125,7 +125,6 @@ html_theme_options = { 'link': '#08f', 'link_hover': 'red', 'extra_nav_links': collections.OrderedDict([ - ('ClickHouse Meetup Berlin', 'https://events.yandex.com/events/meetings/05-10-2017/'), ('Switch to Russian ', '#ru'), ('Single page documentation', '/docs/en/single/'), ('Website home', '/'), diff --git a/website/index.css b/website/index.css index 3f72e172b83..47afa8bfd01 100644 --- a/website/index.css +++ b/website/index.css @@ -219,7 +219,7 @@ a:hover, a:active { #announcement-link { color: #000; text-decoration: underline; - font: 400 200%/133% 'Yandex Sans Display Web',Arial,sans-serif; + font: 400 150%/133% 'Yandex Sans Display Web',Arial,sans-serif; margin: 1em 0; } #announcement-link:hover { diff --git a/website/index.html b/website/index.html index a83c123da27..2787a11b008 100644 --- a/website/index.html +++ b/website/index.html @@ -92,7 +92,7 @@ From 8bebe667a30aa9c7164b915aa59416157fcdb447 Mon Sep 17 00:00:00 2001 From: AndreevDm Date: Wed, 11 Oct 2017 16:30:24 +0300 Subject: [PATCH 04/14] Update CHANGELOG_RU.md --- CHANGELOG_RU.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG_RU.md b/CHANGELOG_RU.md index 97dce55a89e..724fb015166 100644 --- a/CHANGELOG_RU.md +++ b/CHANGELOG_RU.md @@ -77,7 +77,7 @@ * Добавлен параметр max_size для агрегатной функции `groupArray(max_size)(column)`, и оптимизирована её производительность ## Основные изменения: -* Улучшение безопасности: все файлы сервера создаются с правами 0640 +* Улучшение безопасности: все файлы сервера создаются с правами 0640 (можно поменять, через параметр в конфиге) * Улучшены сообщения об ошибках в случае синтаксически неверных запросов * Значительно уменьшен расход оперативной памяти и улучшена производительность слияний больших MergeTree-кусков данных * Значительно увеличена производительность слияний данных для движка ReplacingMergeTree From 09828ed597b9a5dd1a7b0ff10e386e9d48ac0258 Mon Sep 17 00:00:00 2001 From: AndreevDm Date: Wed, 11 Oct 2017 16:35:30 +0300 Subject: [PATCH 05/14] Update CHANGELOG.md --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ef7ff17d2ee..b9c71f69f62 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -79,7 +79,7 @@ This is a bugfix release. The following bugs were fixed: ## Major changes: -* Improved security: all server files are created with 0640 permissions. +* Improved security: all server files are created with 0640 permissions (can be changed via config parameter). * Improved error messages for queries with invalid syntax. * Significantly reduced memory consumption and improved performance when merging large sections of MergeTree data. * Significantly increased the performance of data merges for the ReplacingMergeTree engine. From a68521e016de5f03ad9d9271b381979125de575a Mon Sep 17 00:00:00 2001 From: AndreevDm Date: Wed, 11 Oct 2017 16:36:05 +0300 Subject: [PATCH 06/14] Update CHANGELOG_RU.md --- CHANGELOG_RU.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG_RU.md b/CHANGELOG_RU.md index 724fb015166..f48d5e43a4c 100644 --- a/CHANGELOG_RU.md +++ b/CHANGELOG_RU.md @@ -77,7 +77,7 @@ * Добавлен параметр max_size для агрегатной функции `groupArray(max_size)(column)`, и оптимизирована её производительность ## Основные изменения: -* Улучшение безопасности: все файлы сервера создаются с правами 0640 (можно поменять, через параметр в конфиге) +* Улучшение безопасности: все файлы сервера создаются с правами 0640 (можно поменять, через параметр в конфиге). * Улучшены сообщения об ошибках в случае синтаксически неверных запросов * Значительно уменьшен расход оперативной памяти и улучшена производительность слияний больших MergeTree-кусков данных * Значительно увеличена производительность слияний данных для движка ReplacingMergeTree From 4814eb70722e3b5495f7793836dfb98de9c3c8c0 Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Wed, 11 Oct 2017 17:41:49 +0300 Subject: [PATCH 07/14] Whitespaces [#CLICKHOUSE-2]. --- dbms/src/Storages/System/StorageSystemTables.cpp | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/dbms/src/Storages/System/StorageSystemTables.cpp b/dbms/src/Storages/System/StorageSystemTables.cpp index f39186487ac..a1574e00e00 100644 --- a/dbms/src/Storages/System/StorageSystemTables.cpp +++ b/dbms/src/Storages/System/StorageSystemTables.cpp @@ -22,10 +22,10 @@ StorageSystemTables::StorageSystemTables(const std::string & name_) : name(name_), columns { - {"database", std::make_shared()}, - {"name", std::make_shared()}, - {"engine", std::make_shared()}, - {"metadata_modification_time", std::make_shared()} + {"database", std::make_shared()}, + {"name", std::make_shared()}, + {"engine", std::make_shared()}, + {"metadata_modification_time", std::make_shared()} } { } From a7f46a4e663927707266ecabd56b2f4c41ce8745 Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Wed, 11 Oct 2017 17:44:12 +0300 Subject: [PATCH 08/14] Implemented TODO [#CLICKHOUSE-2]. --- debian/clickhouse-server.init | 6 ------ 1 file changed, 6 deletions(-) diff --git a/debian/clickhouse-server.init b/debian/clickhouse-server.init index 9e51bf417c2..81a158a59f0 100755 --- a/debian/clickhouse-server.init +++ b/debian/clickhouse-server.init @@ -138,12 +138,6 @@ initdb() echo "Changing owner of [${CLICKHOUSE_LOGDIR}] to [${CLICKHOUSE_LOGDIR_USER}:${CLICKHOUSE_GROUP}]" chown ${CLICKHOUSE_LOGDIR_USER}:${CLICKHOUSE_GROUP} ${CLICKHOUSE_LOGDIR} fi - - # Temporary fix for old metrika user, remove after 2017-06-01 - if [ ! -z ${CLICKHOUSE_DATADIR_OLD} ] && [ -d ${CLICKHOUSE_DATADIR_OLD} ] && ! $(su -s $SHELL ${CLICKHOUSE_USER} -c "test -w ${CLICKHOUSE_DATADIR_OLD}") ; then - echo "Changing owner of old [${CLICKHOUSE_DATADIR_OLD}] to [${CLICKHOUSE_USER}:${CLICKHOUSE_GROUP}]" - chown -RL ${CLICKHOUSE_USER}:${CLICKHOUSE_GROUP} ${CLICKHOUSE_DATADIR_OLD} - fi } From 459d52a37140b231d2bbd24dc2e096735823348d Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Wed, 11 Oct 2017 18:00:34 +0300 Subject: [PATCH 09/14] Whitespaces [#CLICKHOUSE-2]. --- debian/clickhouse-server-base.postinst | 108 ++++++++++++------------- 1 file changed, 54 insertions(+), 54 deletions(-) diff --git a/debian/clickhouse-server-base.postinst b/debian/clickhouse-server-base.postinst index 3dc07fcb384..d5a7d8bad9f 100644 --- a/debian/clickhouse-server-base.postinst +++ b/debian/clickhouse-server-base.postinst @@ -11,74 +11,74 @@ CLICKHOUSE_SERVER_ETCDIR=/etc/clickhouse-server if [ "$1" = configure ]; then - if [ -x "/etc/init.d/clickhouse-server" ]; then - update-rc.d clickhouse-server defaults 19 19 >/dev/null || exit $? - fi + if [ -x "/etc/init.d/clickhouse-server" ]; then + update-rc.d clickhouse-server defaults 19 19 >/dev/null || exit $? + fi - # Make sure the administrative user exists - if ! getent passwd ${CLICKHOUSE_USER} > /dev/null; then - adduser --system --disabled-login --no-create-home --home /nonexistent \ - --shell /bin/false --group --gecos "Clickhouse server" clickhouse > /dev/null - fi + # Make sure the administrative user exists + if ! getent passwd ${CLICKHOUSE_USER} > /dev/null; then + adduser --system --disabled-login --no-create-home --home /nonexistent \ + --shell /bin/false --group --gecos "Clickhouse server" clickhouse > /dev/null + fi - # if the user was created manually, make sure the group is there as well - if ! getent group ${CLICKHOUSE_GROUP} > /dev/null; then - addgroup --system ${CLICKHOUSE_GROUP} > /dev/null - fi + # if the user was created manually, make sure the group is there as well + if ! getent group ${CLICKHOUSE_GROUP} > /dev/null; then + addgroup --system ${CLICKHOUSE_GROUP} > /dev/null + fi - # make sure user is in the correct group - if ! id -Gn ${CLICKHOUSE_USER} | grep -qw ${CLICKHOUSE_USER}; then - adduser ${CLICKHOUSE_USER} ${CLICKHOUSE_GROUP} > /dev/null - fi + # make sure user is in the correct group + if ! id -Gn ${CLICKHOUSE_USER} | grep -qw ${CLICKHOUSE_USER}; then + adduser ${CLICKHOUSE_USER} ${CLICKHOUSE_GROUP} > /dev/null + fi - # check validity of user and group - if [ "`id -u ${CLICKHOUSE_USER}`" -eq 0 ]; then - echo "The ${CLICKHOUSE_USER} system user must not have uid 0 (root). + # check validity of user and group + if [ "`id -u ${CLICKHOUSE_USER}`" -eq 0 ]; then + echo "The ${CLICKHOUSE_USER} system user must not have uid 0 (root). Please fix this and reinstall this package." >&2 - exit 1 - fi + exit 1 + fi - if [ "`id -g ${CLICKHOUSE_GROUP}`" -eq 0 ]; then - echo "The ${CLICKHOUSE_USER} system user must not have root as primary group. + if [ "`id -g ${CLICKHOUSE_GROUP}`" -eq 0 ]; then + echo "The ${CLICKHOUSE_USER} system user must not have root as primary group. Please fix this and reinstall this package." >&2 - exit 1 - fi + exit 1 + fi - if [ ! -d ${CLICKHOUSE_DATADIR} ]; then - # only for compatibility for old /opt/clickhouse, remove after 2017-06-01 - if [ -d ${CLICKHOUSE_DATADIR_OLD} ]; then - ln -s ${CLICKHOUSE_DATADIR_OLD} ${CLICKHOUSE_DATADIR} - else - # DONT remove after 2017-06-01 : - mkdir -p ${CLICKHOUSE_DATADIR} - chown ${CLICKHOUSE_USER}:${CLICKHOUSE_GROUP} ${CLICKHOUSE_DATADIR} - chmod 700 ${CLICKHOUSE_DATADIR} - fi - fi + if [ ! -d ${CLICKHOUSE_DATADIR} ]; then + # only for compatibility for old /opt/clickhouse, remove after 2017-06-01 + if [ -d ${CLICKHOUSE_DATADIR_OLD} ]; then + ln -s ${CLICKHOUSE_DATADIR_OLD} ${CLICKHOUSE_DATADIR} + else + # DONT remove after 2017-06-01 : + mkdir -p ${CLICKHOUSE_DATADIR} + chown ${CLICKHOUSE_USER}:${CLICKHOUSE_GROUP} ${CLICKHOUSE_DATADIR} + chmod 700 ${CLICKHOUSE_DATADIR} + fi + fi - if [ ! -d ${CLICKHOUSE_LOGDIR} ]; then - mkdir -p ${CLICKHOUSE_LOGDIR} - chown root:${CLICKHOUSE_GROUP} ${CLICKHOUSE_LOGDIR} - # Allow everyone to read logs, root and clickhouse to read-write - chmod 775 ${CLICKHOUSE_LOGDIR} - fi + if [ ! -d ${CLICKHOUSE_LOGDIR} ]; then + mkdir -p ${CLICKHOUSE_LOGDIR} + chown root:${CLICKHOUSE_GROUP} ${CLICKHOUSE_LOGDIR} + # Allow everyone to read logs, root and clickhouse to read-write + chmod 775 ${CLICKHOUSE_LOGDIR} + fi - if [ -d ${CLICKHOUSE_LOGDIR} ]; then - # only for compatibility for old metrika user, remove string after 2017-06-01 : - su -s /bin/sh ${CLICKHOUSE_USER} -c "test -w ${CLICKHOUSE_LOGDIR}" || chown -R root:${CLICKHOUSE_GROUP} ${CLICKHOUSE_LOGDIR}; chmod -R ug+rw ${CLICKHOUSE_LOGDIR} - fi + if [ -d ${CLICKHOUSE_LOGDIR} ]; then + # only for compatibility for old metrika user, remove string after 2017-06-01 : + su -s /bin/sh ${CLICKHOUSE_USER} -c "test -w ${CLICKHOUSE_LOGDIR}" || chown -R root:${CLICKHOUSE_GROUP} ${CLICKHOUSE_LOGDIR}; chmod -R ug+rw ${CLICKHOUSE_LOGDIR} + fi - if [ -d ${CLICKHOUSE_SERVER_ETCDIR} ]; then - # -R only for compatibility for old metrika user, remove -R after 2017-06-01 - su -s /bin/sh ${CLICKHOUSE_USER} -c "test -w ${CLICKHOUSE_SERVER_ETCDIR}" || chown -R ${CLICKHOUSE_USER}:${CLICKHOUSE_GROUP} ${CLICKHOUSE_SERVER_ETCDIR} - fi + if [ -d ${CLICKHOUSE_SERVER_ETCDIR} ]; then + # -R only for compatibility for old metrika user, remove -R after 2017-06-01 + su -s /bin/sh ${CLICKHOUSE_USER} -c "test -w ${CLICKHOUSE_SERVER_ETCDIR}" || chown -R ${CLICKHOUSE_USER}:${CLICKHOUSE_GROUP} ${CLICKHOUSE_SERVER_ETCDIR} + fi - # Clean old dynamic compilation results - if [ -d "${CLICKHOUSE_DATADIR}/build" ]; then - rm -f ${CLICKHOUSE_DATADIR}/build/*.cpp ${CLICKHOUSE_DATADIR}/build/*.so ||: - fi + # Clean old dynamic compilation results + if [ -d "${CLICKHOUSE_DATADIR}/build" ]; then + rm -f ${CLICKHOUSE_DATADIR}/build/*.cpp ${CLICKHOUSE_DATADIR}/build/*.so ||: + fi fi From f42ae974d16c241ed54e7bf0e41875cf32618298 Mon Sep 17 00:00:00 2001 From: proller Date: Wed, 11 Oct 2017 21:22:28 +0300 Subject: [PATCH 10/14] Remove wrong test --- .../0_stateless/00507_nullable.reference | 2 - .../queries/0_stateless/00507_nullable.sql | 41 ------------------- 2 files changed, 43 deletions(-) delete mode 100644 dbms/tests/queries/0_stateless/00507_nullable.reference delete mode 100644 dbms/tests/queries/0_stateless/00507_nullable.sql diff --git a/dbms/tests/queries/0_stateless/00507_nullable.reference b/dbms/tests/queries/0_stateless/00507_nullable.reference deleted file mode 100644 index 1c42f9991e2..00000000000 --- a/dbms/tests/queries/0_stateless/00507_nullable.reference +++ /dev/null @@ -1,2 +0,0 @@ -1970-01-02 2 0 FAILED 2 -still alive diff --git a/dbms/tests/queries/0_stateless/00507_nullable.sql b/dbms/tests/queries/0_stateless/00507_nullable.sql deleted file mode 100644 index 46e7aef5614..00000000000 --- a/dbms/tests/queries/0_stateless/00507_nullable.sql +++ /dev/null @@ -1,41 +0,0 @@ -CREATE DATABASE IF NOT EXISTS test; -DROP TABLE IF EXISTS test.runs; -DROP TABLE IF EXISTS test.tests; - -CREATE TABLE test.runs -( - date Date, - id UInt64, - t_id UInt64, - status Enum8('OK' = 1, 'FAILED' = 2, 'SKIPPED' = 3, 'DISCOVERED' = 4), - run_id UInt64 DEFAULT id -) ENGINE = MergeTree(date, (t_id, id), 8192); - -CREATE TABLE test.tests -( - date Date, - id UInt64, - path Nullable(String), - suite_id Nullable(String) -) ENGINE = MergeTree(date, id, 8192); - -INSERT INTO test.tests (date, id) VALUES (1,1); -INSERT INTO test.runs (date, id) VALUES (1,1); -INSERT INTO test.runs (date, id, status) VALUES (1,2, 'FAILED'); -INSERT INTO test.tests (date, id, path) VALUES (1,2 ,'rtline1'); - -SELECT * -FROM test.runs AS r -WHERE (r.status = 'FAILED') AND ( -( - SELECT path - FROM test.tests AS t - WHERE t.id = r.id - LIMIT 1 -) LIKE 'rtline%') -LIMIT 1; - -SELECT 'still alive'; - -DROP TABLE test.runs; -DROP TABLE test.tests; From 27307685f6ad29aa0bd11234650fe16645b590cc Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Wed, 11 Oct 2017 22:07:03 +0300 Subject: [PATCH 11/14] Implemented TODO [#CLICKHOUSE-2]. --- debian/clickhouse-server-base.postinst | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-) diff --git a/debian/clickhouse-server-base.postinst b/debian/clickhouse-server-base.postinst index d5a7d8bad9f..7dd4925c8e5 100644 --- a/debian/clickhouse-server-base.postinst +++ b/debian/clickhouse-server-base.postinst @@ -4,7 +4,6 @@ set -e CLICKHOUSE_USER=clickhouse CLICKHOUSE_GROUP=${CLICKHOUSE_USER} CLICKHOUSE_DATADIR=/var/lib/clickhouse -CLICKHOUSE_DATADIR_OLD=/opt/clickhouse # remove after 2017-06-01 CLICKHOUSE_LOGDIR=/var/log/clickhouse-server CLICKHOUSE_SERVER_ETCDIR=/etc/clickhouse-server @@ -45,15 +44,9 @@ Please fix this and reinstall this package." >&2 fi if [ ! -d ${CLICKHOUSE_DATADIR} ]; then - # only for compatibility for old /opt/clickhouse, remove after 2017-06-01 - if [ -d ${CLICKHOUSE_DATADIR_OLD} ]; then - ln -s ${CLICKHOUSE_DATADIR_OLD} ${CLICKHOUSE_DATADIR} - else - # DONT remove after 2017-06-01 : - mkdir -p ${CLICKHOUSE_DATADIR} - chown ${CLICKHOUSE_USER}:${CLICKHOUSE_GROUP} ${CLICKHOUSE_DATADIR} - chmod 700 ${CLICKHOUSE_DATADIR} - fi + mkdir -p ${CLICKHOUSE_DATADIR} + chown ${CLICKHOUSE_USER}:${CLICKHOUSE_GROUP} ${CLICKHOUSE_DATADIR} + chmod 700 ${CLICKHOUSE_DATADIR} fi if [ ! -d ${CLICKHOUSE_LOGDIR} ]; then @@ -65,7 +58,7 @@ Please fix this and reinstall this package." >&2 if [ -d ${CLICKHOUSE_LOGDIR} ]; then - # only for compatibility for old metrika user, remove string after 2017-06-01 : + # only for compatibility for old metrika user, remove string after 2017-06-01 su -s /bin/sh ${CLICKHOUSE_USER} -c "test -w ${CLICKHOUSE_LOGDIR}" || chown -R root:${CLICKHOUSE_GROUP} ${CLICKHOUSE_LOGDIR}; chmod -R ug+rw ${CLICKHOUSE_LOGDIR} fi From 041623dee166e9c4dda1b6c45cdfbe532c7031dc Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Wed, 11 Oct 2017 22:11:51 +0300 Subject: [PATCH 12/14] Moved chown to correct place [#CLICKHOUSE-2]. --- debian/clickhouse-server-base.postinst | 7 ------- debian/clickhouse-server.init | 4 ++++ 2 files changed, 4 insertions(+), 7 deletions(-) diff --git a/debian/clickhouse-server-base.postinst b/debian/clickhouse-server-base.postinst index 7dd4925c8e5..476dccfd271 100644 --- a/debian/clickhouse-server-base.postinst +++ b/debian/clickhouse-server-base.postinst @@ -5,7 +5,6 @@ CLICKHOUSE_USER=clickhouse CLICKHOUSE_GROUP=${CLICKHOUSE_USER} CLICKHOUSE_DATADIR=/var/lib/clickhouse CLICKHOUSE_LOGDIR=/var/log/clickhouse-server -CLICKHOUSE_SERVER_ETCDIR=/etc/clickhouse-server if [ "$1" = configure ]; then @@ -63,12 +62,6 @@ Please fix this and reinstall this package." >&2 fi - if [ -d ${CLICKHOUSE_SERVER_ETCDIR} ]; then - # -R only for compatibility for old metrika user, remove -R after 2017-06-01 - su -s /bin/sh ${CLICKHOUSE_USER} -c "test -w ${CLICKHOUSE_SERVER_ETCDIR}" || chown -R ${CLICKHOUSE_USER}:${CLICKHOUSE_GROUP} ${CLICKHOUSE_SERVER_ETCDIR} - fi - - # Clean old dynamic compilation results if [ -d "${CLICKHOUSE_DATADIR}/build" ]; then rm -f ${CLICKHOUSE_DATADIR}/build/*.cpp ${CLICKHOUSE_DATADIR}/build/*.so ||: diff --git a/debian/clickhouse-server.init b/debian/clickhouse-server.init index 81a158a59f0..3261825b74b 100755 --- a/debian/clickhouse-server.init +++ b/debian/clickhouse-server.init @@ -95,6 +95,10 @@ check_config() initdb() { + if [ -d ${SYSCONFDIR} ]; then + su -s /bin/sh ${CLICKHOUSE_USER} -c "test -w ${SYSCONFDIR}" || chown ${CLICKHOUSE_USER}:${CLICKHOUSE_GROUP} ${SYSCONFDIR} + fi + if [ -x "$BINDIR/$GENERIC_PROGRAM" ]; then CLICKHOUSE_DATADIR_FROM_CONFIG=$(su -s $SHELL ${CLICKHOUSE_USER} -c "$BINDIR/$GENERIC_PROGRAM extract-from-config --config-file=\"$CLICKHOUSE_CONFIG\" --key=path") if [ "(" "$?" -ne "0" ")" -o "(" -z "${CLICKHOUSE_DATADIR_FROM_CONFIG}" ")" ]; then From bc4a78f168854fac4546d5a7130d078301003e1a Mon Sep 17 00:00:00 2001 From: Mikhail Filimonov Date: Wed, 11 Oct 2017 21:45:06 +0200 Subject: [PATCH 13/14] Fixing documentation fragment about "default" user. It is not mandatory anymore. --- docs/en/operations/access_rights.rst | 4 +++- docs/ru/operations/access_rights.rst | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/en/operations/access_rights.rst b/docs/en/operations/access_rights.rst index a2e0c8da2cb..020bb64f5f5 100644 --- a/docs/en/operations/access_rights.rst +++ b/docs/en/operations/access_rights.rst @@ -40,7 +40,9 @@ Users are recorded in the ``users`` section. Let's look at part of the ``users.x Here we can see that two users are declared: ``default`` and ``web``. We added the ``web`` user ourselves. -The ``default`` user is chosen in cases when the username is not passed, so this user must be present in the config file. The ``default`` user is also used for distributed query processing - the system accesses remote servers under this username. So the ``default`` user must have an empty password and must not have substantial restrictions or quotas - otherwise, distributed queries will fail. +The ``default`` user is chosen in cases when the username is not passed. The ``default`` user can also be used for distributed query processing - the system accesses remote servers using this username if no ``user`` and ``password`` were configured for that server inside cluster configuration (see also section about "Distributed" table engine). + +For connection to the server inside cluster you should use the user without any substantial restrictions or quotas - otherwise, distributed queries will fail. The password is specified in plain text directly in the config. In this regard, you should not consider these passwords as providing security against potential malicious attacks. Rather, they are necessary for protection from Yandex employees. diff --git a/docs/ru/operations/access_rights.rst b/docs/ru/operations/access_rights.rst index 4078b357252..83bb19c7808 100644 --- a/docs/ru/operations/access_rights.rst +++ b/docs/ru/operations/access_rights.rst @@ -60,7 +60,9 @@ Здесь видно объявление двух пользователей - ``default`` и ``web``. Пользователя ``web`` мы добавили самостоятельно. -Пользователь ``default`` выбирается в случаях, когда имя пользователя не передаётся, поэтому такой пользователь должен присутствовать в конфигурационном файле обязательно. Также пользователь ``default`` используется при распределённой обработки запроса - система ходит на удалённые серверы под ним. Поэтому, у пользователя ``default`` должен быть пустой пароль и не должно быть выставлено существенных ограничений или квот - иначе распределённые запросы сломаются. +Пользователь ``default`` выбирается в случаях, когда имя пользователя не передаётся. Также пользователь ``default`` может использоваться при распределённой обработке запроса - если в конфигурации кластера для сервера не указаны ``user`` и ``password``. (см. раздел о движке "Distributed"). + +Пользователь, который используется для обмена информацией между серверами, объединенными в кластер, не должен иметь существенных ограничений или квот - иначе распределённые запросы сломаются. Пароль указывается либо в открытом виде (не рекомендуется), либо в виде SHA-256. Хэш не содержит соль. В связи с этим, не следует рассматривать такие пароли, как защиту от потенциального злоумышленника. Скорее, они нужны для защиты от сотрудников. From dfb64cf0c0e7ceff288f5a2e2fa72806ecaa0a5c Mon Sep 17 00:00:00 2001 From: Yuri Dyachenko Date: Wed, 11 Oct 2017 23:38:56 +0300 Subject: [PATCH 14/14] append yurial/clickhouse-client to docs/interfaces --- docs/en/interfaces/third-party_client_libraries.rst | 1 + docs/ru/interfaces/third-party_client_libraries.rst | 1 + 2 files changed, 2 insertions(+) diff --git a/docs/en/interfaces/third-party_client_libraries.rst b/docs/en/interfaces/third-party_client_libraries.rst index 0ee86db23f7..cf87beb66f2 100644 --- a/docs/en/interfaces/third-party_client_libraries.rst +++ b/docs/en/interfaces/third-party_client_libraries.rst @@ -7,6 +7,7 @@ There exist third-party client libraries for ClickHouse: - `infi.clickhouse_orm `_ - `sqlalchemy-clickhouse `_ - `clickhouse-driver `_ + - `clickhouse-client `_ * PHP - `clickhouse-php-client `_ - `PhpClickHouseClient `_ diff --git a/docs/ru/interfaces/third-party_client_libraries.rst b/docs/ru/interfaces/third-party_client_libraries.rst index 7460e2300c0..d35207965e0 100644 --- a/docs/ru/interfaces/third-party_client_libraries.rst +++ b/docs/ru/interfaces/third-party_client_libraries.rst @@ -7,6 +7,7 @@ - `infi.clickhouse_orm `_ - `sqlalchemy-clickhouse `_ - `clickhouse-driver `_ + - `clickhouse-client `_ * PHP - `clickhouse-php-client `_ - `PhpClickHouseClient `_