WIP on docs (#3860)

* CLICKHOUSE-4063: less manual html @ index.md

* CLICKHOUSE-4063: recommend markdown="1" in README.md

* CLICKHOUSE-4003: manually purge custom.css for now

* CLICKHOUSE-4064: expand <details> before any print (including to pdf)

* CLICKHOUSE-3927: rearrange interfaces/formats.md a bit

* CLICKHOUSE-3306: add few http headers

* Remove copy-paste introduced in #3392

* Hopefully better chinese fonts #3392

* get rid of tabs @ custom.css

* Apply comments and patch from #3384

* Add jdbc.md to ToC and some translation, though it still looks badly incomplete

* minor punctuation

* Add some backlinks to official website from mirrors that just blindly take markdown sources

* Do not make fonts extra light

* find . -name '*.md' -type f | xargs -I{} perl -pi -e 's//g' {}

* find . -name '*.md' -type f | xargs -I{} perl -pi -e 's/ sql/g' {}

* Remove outdated stuff from roadmap.md

* Not so light font on front page too

* Refactor Chinese formats.md to match recent changes in other languages

* Update some links on front page

* Remove some outdated comment

* Add twitter link to front page

* More front page links tuning

* Add Amsterdam meetup link

* Smaller font to avoid second line

* Add Amsterdam link to README.md

* Proper docs nav translation

* Back to 300 font-weight except Chinese

* fix docs build

* Update Amsterdam link

* remove symlinks

* more zh punctuation

* apply lost comment by @zhang2014

* Apply comments by @zhang2014 from #3417

* Remove Beijing link

* rm incorrect symlink

* restore content of docs/zh/operations/table_engines/index.md

* CLICKHOUSE-3751: stem terms while searching docs

* CLICKHOUSE-3751: use English stemmer in non-English docs too

* CLICKHOUSE-4135 fix

* Remove past meetup link

* Add blog link to top nav

* Add ContentSquare article link

* Add form link to front page + refactor some texts

* couple markup fixes

* minor

* Introduce basic ODBC driver page in docs

* More verbose 3rd party libs disclaimer

* Put third-party stuff into a separate folder

* Separate third-party stuff in ToC too

* Update links

* Move stuff that is not really (only) a client library into a separate page

* Add clickhouse-hdfs-loader link

* Some introduction for "interfaces" section

* Rewrite tcp.md

* http_interface.md -> http.md

* fix link

* Remove unconvenient error for now

* try to guess anchor instead of failing

* remove symlink

* Remove outdated info from introduction

* remove ru roadmap.md

* replace ru roadmap.md with symlink

* Update roadmap.md

* lost file

* Title case in toc_en.yml

* Sync "Functions" ToC section with en

* Remove reference to pretty old ClickHouse release from docs

* couple lost symlinks in fa

* Close quote in proper place

* Rewrite en/getting_started/index.md

* Sync en<>ru getting_started/index.md

* minor changes

* Some gui.md refactoring

* Translate DataGrip section to ru

* Translate DataGrip section to zh

* Translate DataGrip section to fa

* Translate DBeaver section to fa

* Translate DBeaver section to zh

* Split third-party GUI to open-source and commercial

* Mention some RDBMS integrations + ad-hoc translation fixes

* Add rel="external nofollow" to outgoing links from docs

* Lost blank lines

* Fix class name

* More rel="external nofollow"

* Apply suggestions by @sundy-li

* Mobile version of front page improvements

* test

* test 2

* test 3

* Update LICENSE

* minor docs fix

* Highlight current article as suggested by @sundy-li

* fix link destination

* Introduce backup.md (only "en" for now)

* Mention INSERT+SELECT in backup.md

* Some improvements for replication.md

* Add backup.md to toc

* Mention clickhouse-backup tool

* Mention LightHouse in third-party GUI list

* Introduce interfaces/third-party/proxy.md

* Add clickhouse-bulk to proxy.md

* Major extension of integrations.md contents

* fix link target

* remove unneeded file

* better toc item name

* fix markdown

* better ru punctuation

* Add yet another possible backup approach

* Simplify copying permalinks to headers

* Support non-eng link anchors in docs + update some deps

* Generate anchors for single-page mode automatically

* Remove anchors to top of pages

* Remove anchors that nobody links to

* build fixes

* fix few links

* restore css

* fix some links

* restore gifs

* fix lost words

* more docs fixes

* docs fixes

* NULL anchor

* update urllib3 dependency

* more fixes

* Remove excessive content from print version

* Try short license again

* Back to long license for now

* Introduce anchor integrity checks for single-page docs

* Add --save-raw-single-page option to build.py (helps to debug incorrect anchors)

* fix kafka engine links

* fix one class of broken anchors

* fix some broken links

* Add https://github.com/hatarist/clickhouse-cli to third-party section (in gui.md for now, maybe will add cli.md later)

* fix one more class of links to nowhere

* less duplicate anchors

* get rid of weird anchors

* fix anchor

* fix link

* fix couple links

CHANGELOG.md

@@ -926,7 +926,7 @@ This release contains bug fixes for the previous release 1.1.54310:
 ### New features:
 
 * Custom partitioning key for the MergeTree family of table engines.
-* [ Kafka](https://clickhouse.yandex/docs/en/single/index.html#document-table_engines/kafka)  table engine.
+* [Kafka](https://clickhouse.yandex/docs/en/operations/table_engines/kafka/)  table engine.
 * Added support for loading [CatBoost](https://catboost.yandex/)  models and applying them to data stored in ClickHouse.
 * Added support for time zones with non-integer offsets from UTC.
 * Added support for arithmetic operations with time intervals.

CHANGELOG_RU.md

@@ -988,7 +988,7 @@
 
 ### Новые возможности:
 * Произвольный ключ партиционирования для таблиц семейства MergeTree.
-* Движок таблиц [Kafka](https://clickhouse.yandex/docs/en/single/index.html#document-table_engines/kafka).
+* Движок таблиц [Kafka](https://clickhouse.yandex/docs/en/operations/table_engines/kafka/).
 * Возможность загружать модели [CatBoost](https://catboost.yandex/) и применять их к данным, хранящимся в ClickHouse.
 * Поддержка часовых поясов с нецелым смещением от UTC.
 * Поддержка операций с временными интервалами.

docs/en/interfaces/third-party/gui.md

@@ -48,6 +48,16 @@ Features:
 - Table preview with filtering and sorting.
 - Read-only queries execution.
 
+### clickhouse-cli
+
+[clickhouse-cli](https://github.com/hatarist/clickhouse-cli) is an alternative command line client for ClickHouse, written in Python 3.
+
+Features:
+- Autocompletion.
+- Syntax highlighting for the queries and data output.
+- Pager support for the data output.
+- Custom PostgreSQL-like commands.
+
 ## Commercial
 
 ### DBeaver

docs/en/operations/access_rights.md

@@ -96,7 +96,7 @@ Access to the `system` database is always allowed (since this database is used f
 
 The user can get a list of all databases and tables in them by using `SHOW` queries or system tables, even if access to individual databases isn't allowed.
 
-Database access is not related to the [readonly](settings/query_complexity.md#query_complexity_readonly) setting. You can't grant full access to one database and `readonly` access to another one.
+Database access is not related to the [readonly](settings/query_complexity.md#readonly) setting. You can't grant full access to one database and `readonly` access to another one.
 
 
 [Original article](https://clickhouse.yandex/docs/en/operations/access_rights/) <!--hide-->

docs/en/operations/server_settings/settings.md

@@ -74,7 +74,7 @@ To get a list of databases, use the [SHOW DATABASES](../../query_language/misc.m
 
 Default settings profile.
 
-Settings profiles are located in the file specified in the parameter [user_config](#user-config).
+Settings profiles are located in the file specified in the parameter `user_config`.
 
 **Example**
 
@@ -196,7 +196,7 @@ For more details, see [GraphiteMergeTree](../../operations/table_engines/graphit
 
 The port for connecting to the server over HTTP(s).
 
-If `https_port` is specified, [openSSL]() must be configured.
+If `https_port` is specified, [openSSL](#openssl) must be configured.
 
 If `http_port` is specified, the openSSL configuration is ignored even if it is set.
 

docs/en/operations/table_engines/url.md

@@ -3,7 +3,7 @@
 # URL(URL, Format)
 
 Manages data on a remote HTTP/HTTPS server. This engine is similar
-to the [`File`](./file.md#) engine.
+to the [File](file.md) engine.
 
 ## Using the engine in the ClickHouse server
 

docs/en/operations/utils/clickhouse-local.md

@@ -3,7 +3,7 @@
 
 The `clickhouse-local` program enables you to perform fast processing on local files, without having to deploy and configure the ClickHouse server.
 
-Accepts data that represent tables and queries them using [ClickHouse SQL dialect](../../query_language/index.md#queries).
+Accepts data that represent tables and queries them using [ClickHouse SQL dialect](../../query_language/index.md).
 
 `clickhouse-local` uses the same core as ClickHouse server, so it supports most of the features and the same set of formats and table engines.
 

docs/en/query_language/dicts/index.md

@@ -8,8 +8,8 @@ ClickHouse supports special functions for working with dictionaries that can be
 
 ClickHouse supports:
 
-- [Built-in dictionaries](internal_dicts.md#internal_dicts) with a specific [set of functions](../functions/ym_dict_functions.md#ym_dict_functions).
-- [Plug-in (external) dictionaries](external_dicts.md) with a [set of functions](../functions/ext_dict_functions.md#ext_dict_functions).
+- [Built-in dictionaries](internal_dicts.md#internal_dicts) with a specific [set of functions](../functions/ym_dict_functions.md).
+- [Plug-in (external) dictionaries](external_dicts.md) with a [set of functions](../functions/ext_dict_functions.md).
 
 
 [Original article](https://clickhouse.yandex/docs/en/query_language/dicts/) <!--hide-->

docs/fa/interfaces/third-party/gui.md

@@ -49,6 +49,16 @@ interface تحت وب برای ClickHouse در پروژه [Tabix](https://github
 - پیش نمایش جدول با فیلتر کردن و مرتب سازی.
 - اعداد نمایش داده شده فقط خواندنی
 
+### clickhouse-cli
+
+[clickhouse-cli](https://github.com/hatarist/clickhouse-cli) یک مشتری خط فرمان برای ClickHouse است که در پایتون 3 نوشته شده است.
+
+امکانات:
+- تکمیل خودکار
+- نحو برجسته برای نمایش داده ها و خروجی داده ها.
+- پشتیبانی از Pager برای خروجی داده.
+- دستورات پست سفارشی مانند PostgreSQL.
+
 ## تجاری
 
 ### DBeaver

docs/ru/interfaces/third-party/gui.md

@@ -47,10 +47,20 @@
 
 Основные возможности:
 
-- Список таблиц с фильтрацией и метаданными.
-- Предварительный просмотр таблицы с фильтрацией и сортировкой.
+- Список таблиц с фильтрацией и метаданными;
+- Предварительный просмотр таблицы с фильтрацией и сортировкой;
 - Выполнение запросов только для чтения.
 
+### clickhouse-cli
+
+[clickhouse-cli](https://github.com/hatarist/clickhouse-cli) - это альтернативный клиент командной строки для ClickHouse, написанный на Python 3.
+
+Основные возможности:
+- Автодополнение;
+- Подсветка синтаксиса для запросов и вывода данных;
+- Поддержка постраничного просмотра для результирующих данных;
+- Дополнительные PostgreSQL-подобные команды.
+
 ## Коммерческие
 
 ### DBeaver

docs/ru/operations/access_rights.md

@@ -96,6 +96,6 @@
 
 Пользователь может получить список всех БД и таблиц в них с помощью запросов `SHOW` или системных таблиц, даже если у него нет доступа к отдельным БД.
 
-Доступ к БД не связан с настройкой [readonly](settings/query_complexity.md#query_complexity_readonly). Невозможно дать полный доступ к одной БД и `readonly` к другой.
+Доступ к БД не связан с настройкой [readonly](settings/query_complexity.md#readonly). Невозможно дать полный доступ к одной БД и `readonly` к другой.
 
 [Оригинальная статья](https://clickhouse.yandex/docs/ru/operations/access_rights/) <!--hide-->

docs/ru/operations/server_settings/settings.md

@@ -74,7 +74,7 @@ ClickHouse проверит условия `min_part_size` и `min_part_size_rat
 
 Профиль настроек по умолчанию.
 
-Профили настроек находятся в файле, указанном в параметре [user_config](#user-config).
+Профили настроек находятся в файле, указанном в параметре `user_config`.
 
 **Пример**
 

docs/ru/operations/table_engines/url.md

@@ -3,7 +3,7 @@
 # URL(URL, Format)
 
 Управляет данными на удаленном HTTP/HTTPS сервере. Данный движок похож
-на движок [`File`](./file.md#).
+на движок [File](file.md).
 
 ## Использование движка в сервере ClickHouse
 

docs/ru/query_language/dicts/index.md

@@ -8,7 +8,7 @@ ClickHouse поддерживает специальные функции для
 
 ClickHouse поддерживает:
 
-- [Встроенные словари](internal_dicts.md#internal_dicts) со специфическим [набором функций](../functions/ym_dict_functions.md#ym_dict_functions).
-- [Подключаемые (внешние) словари](external_dicts.md) с [набором функций](../functions/ext_dict_functions.md#ext_dict_functions).
+- [Встроенные словари](internal_dicts.md#internal_dicts) со специфическим [набором функций](../functions/ym_dict_functions.md).
+- [Подключаемые (внешние) словари](external_dicts.md) с [набором функций](../functions/ext_dict_functions.md).
 
 [Оригинальная статья](https://clickhouse.yandex/docs/ru/query_language/dicts/) <!--hide-->

docs/tools/build.py

@@ -22,6 +22,7 @@ from mkdocs.commands import build as mkdocs_build
 
 from concatenate import concatenate
 import mdx_clickhouse
+import test
 
 @contextlib.contextmanager
 def temp_dir():
@@ -183,6 +184,22 @@ def build_single_page_version(lang, args, cfg):
                 logging.debug(' '.join(create_pdf_command))
                 subprocess.check_call(' '.join(create_pdf_command), shell=True)
 
+                with temp_dir() as test_dir:
+                    cfg.load_dict({
+                        'docs_dir': docs_temp_lang,
+                        'site_dir': test_dir,
+                        'extra': {
+                            'single_page': False
+                        },
+                        'nav': [
+                            {cfg.data.get('site_name'): 'single.md'}
+                        ]
+                    })
+                    mkdocs_build.build(cfg)
+                    test.test_single_page(os.path.join(test_dir, 'single', 'index.html'), lang)
+                    if args.save_raw_single_page:
+                        shutil.copytree(test_dir, args.save_raw_single_page)
+
 
 def build_redirects(args):
     lang_re_fragment = args.lang.replace(',', '|')
@@ -212,6 +229,7 @@ if __name__ == '__main__':
     arg_parser.add_argument('--theme-dir', default='mkdocs-material-theme')
     arg_parser.add_argument('--output-dir', default='build')
     arg_parser.add_argument('--skip-single-page', action='store_true')
+    arg_parser.add_argument('--save-raw-single-page', type=str)
     arg_parser.add_argument('--verbose', action='store_true')
 
     args = arg_parser.parse_args()

docs/tools/concatenate.py

@@ -18,9 +18,9 @@ import os
 
 
 def concatenate(lang, docs_path, single_page_file):
-
     proj_config = os.path.join(docs_path, 'toc_%s.yml' % lang)
     lang_path = os.path.join(docs_path, lang)
+    az_re = re.compile(r'[a-z]')
 
     with open(proj_config) as cfg_file:
         files_to_concatenate = []
@@ -38,17 +38,28 @@ def concatenate(lang, docs_path, single_page_file):
 
     for path in files_to_concatenate:
         with open(os.path.join(lang_path, path)) as f:
-            tmp_path = path.replace('.md', '/')
+            anchors = set()
+            tmp_path = path.replace('/index.md', '/').replace('.md', '/')
             prefixes = ['', '../', '../../', '../../../']
             parts = tmp_path.split('/')
-            single_page_file.write('<a name="%s/"></a>\n' % parts[-2])
+            anchors.add(parts[-2] + '/')
+            anchors.add('/'.join(parts[1:]))
+
+            for part in parts[0:-2] if len(parts) > 2 else parts:
+                for prefix in prefixes:
+                    anchor = prefix + tmp_path
+                    if anchor:
+                        anchors.add(anchor)
+                        anchors.add('../' + anchor)
+                        anchors.add('../../' + anchor)
+                tmp_path = tmp_path.replace(part, '..')
+
+            for anchor in anchors:
+                if re.search(az_re, anchor):
+                    single_page_file.write('<a name="%s"></a>\n' % anchor)
+
             single_page_file.write('\n\n')
 
-            for part in parts[0:-2]:
-                for prefix in prefixes:
-                    single_page_file.write('<a name="%s"></a>\n' % (prefix + tmp_path))
-                tmp_path = tmp_path.replace(part, '..')
-                
             for l in f:
                 if l.startswith('#'):
                     l = '#' + l

docs/tools/mdx_clickhouse.py

@@ -22,13 +22,14 @@ class ClickHouseLinkMixin(object):
         if el is not None:
             href = el.get('href') or ''
             is_external = href.startswith('http:') or href.startswith('https:')
-            if is_external and not href.startswith('https://clickhouse.yandex'):
-                el.set('rel', 'external nofollow')
+            if is_external:
+                if not href.startswith('https://clickhouse.yandex'):
+                    el.set('rel', 'external nofollow')
             elif single_page:
                 if '#' in href:
                     el.set('href', '#' + href.split('#', 1)[1])
                 else:
-                    el.set('href', '#' + href.replace('.md', '/'))
+                    el.set('href', '#' + href.replace('/index.md', '/').replace('.md', '/'))
         return el
 
 

docs/tools/mkdocs-material-theme/assets/stylesheets/custom.css

@@ -160,12 +160,20 @@ h1, h2, h3, .md-logo {
 .md-current-lang>svg {
     filter: brightness(96%) grayscale(0%) !important;
 }
+
+
+.headerlink:hover {
+    text-decoration: none;
+}
+
 @media only screen and (min-width: 60em) {
     #md-sidebar-flags {
         display: none;
     }
 }
 
-.headerlink:hover {
-    text-decoration: none;
+@media print {
+    .md-typeset a:after {
+        content: "" !important;
+    }
 }

docs/tools/requirements.txt

@@ -1,6 +1,7 @@
 alabaster==0.7.10
 Babel==2.5.1
 backports-abc==0.5
+beautifulsoup4==4.6.3
 certifi==2017.11.5
 chardet==3.0.4
 click==6.7

docs/tools/test.py

@@ -0,0 +1,46 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+import logging
+import sys
+
+import bs4
+
+def test_single_page(input_path, lang):
+    with open(input_path) as f:
+        soup = bs4.BeautifulSoup(
+            f,
+            features='html.parser'
+        )
+        anchor_points = set()
+        duplicate_anchor_points = 0
+        links_to_nowhere = 0
+        for tag in soup.find_all():
+            for anchor_point in [tag.attrs.get('name'), tag.attrs.get('id')]:
+                if anchor_point:
+                    if anchor_point in anchor_points:
+                        duplicate_anchor_points += 1
+                        logging.info('Duplicate anchor point: %s' % anchor_point)
+                    else:
+                        anchor_points.add(anchor_point)
+        for tag in soup.find_all():
+            href = tag.attrs.get('href')
+            if href and href.startswith('#'):
+                if href[1:] not in anchor_points:
+                    links_to_nowhere += 1
+                    logging.info('Link to nowhere: %s' % href)
+
+        if duplicate_anchor_points:
+            logging.error('Found %d duplicate anchor points' % duplicate_anchor_points)
+        if links_to_nowhere:
+            logging.error('Found %d links to nowhere' % links_to_nowhere)
+
+        assert len(anchor_points) > 10, 'Html parsing is probably broken'
+                
+
+if __name__ == '__main__':
+    logging.basicConfig(
+        level=logging.DEBUG,
+        stream=sys.stderr
+    )
+    test_single_page(sys.argv[1], sys.argv[2])

docs/zh/interfaces/third-party/gui.md

@@ -48,6 +48,16 @@ ClickHouse Web 界面 [Tabix](https://github.com/tabixio/tabix).
  - 带有过滤和排序的表格预览。
  - 只读查询执行。
 
+### clickhouse-cli
+
+[clickhouse-cli](https://github.com/hatarist/clickhouse-cli)是ClickHouse的替代命令行客户端，用Python 3编写。
+
+特征：
+ - 自动完成。
+ - 查询和数据输出的语法高亮显示。
+ - 寻呼机支持数据输出。
+ - 自定义PostgreSQL类命令。
+
 ## 商业
 
 ### DBeaver

docs/zh/operations/access_rights.md

@@ -96,7 +96,7 @@ Access to the `system` database is always allowed (since this database is used f
 
 The user can get a list of all databases and tables in them by using `SHOW` queries or system tables, even if access to individual databases isn't allowed.
 
-Database access is not related to the [readonly](settings/query_complexity.md#query_complexity_readonly) setting. You can't grant full access to one database and `readonly` access to another one.
+Database access is not related to the [readonly](settings/query_complexity.md#readonly) setting. You can't grant full access to one database and `readonly` access to another one.
 
 
 [Original article](https://clickhouse.yandex/docs/en/operations/access_rights/) <!--hide-->

docs/zh/operations/server_settings/settings.md

@@ -74,7 +74,7 @@ To get a list of databases, use the [SHOW DATABASES](../../query_language/misc.m
 
 Default settings profile.
 
-Settings profiles are located in the file specified in the parameter [user_config](#user-config).
+Settings profiles are located in the file specified in the parameter `user_config`.
 
 **Example**
 

docs/zh/operations/table_engines/url.md

@@ -3,7 +3,7 @@
 # URL(URL, Format)
 
 Manages data on a remote HTTP/HTTPS server. This engine is similar
-to the [`File`](./file.md#) engine.
+to the [File](file.md) engine.
 
 ## Using the engine in the ClickHouse server
 

docs/zh/operations/utils/clickhouse-local.md

@@ -3,7 +3,7 @@
 
 The `clickhouse-local` program enables you to perform fast processing on local files, without having to deploy and configure the ClickHouse server.
 
-Accepts data that represent tables and queries them using [ClickHouse SQL dialect](../../query_language/index.md#queries).
+Accepts data that represent tables and queries them using [ClickHouse SQL dialect](../../query_language/index.md).
 
 `clickhouse-local` uses the same core as ClickHouse server, so it supports most of the features and the same set of formats and table engines.