[docs] reorganized F.A.Q. (#11864)

* [docs] reorganized F.A.Q.

* normalize markdown

* support iframes in AMP version

docs/en/faq/general.md

@@ -1,58 +0,0 @@
----
-toc_priority: 78
-toc_title: General Questions
----
-
-# General Questions {#general-questions}
-
-## Why Not Use Something Like MapReduce? {#why-not-use-something-like-mapreduce}
-
-We can refer to systems like MapReduce as distributed computing systems in which the reduce operation is based on distributed sorting. The most common open-source solution in this class is [Apache Hadoop](http://hadoop.apache.org). Yandex uses its in-house solution, YT.
-
-These systems aren’t appropriate for online queries due to their high latency. In other words, they can’t be used as the back-end for a web interface. These types of systems aren’t useful for real-time data updates. Distributed sorting isn’t the best way to perform reduce operations if the result of the operation and all the intermediate results (if there are any) are located in the RAM of a single server, which is usually the case for online queries. In such a case, a hash table is an optimal way to perform reduce operations. A common approach to optimizing map-reduce tasks is pre-aggregation (partial reduce) using a hash table in RAM. The user performs this optimization manually. Distributed sorting is one of the main causes of reduced performance when running simple map-reduce tasks.
-
-Most MapReduce implementations allow you to execute arbitrary code on a cluster. But a declarative query language is better suited to OLAP to run experiments quickly. For example, Hadoop has Hive and Pig. Also consider Cloudera Impala or Shark (outdated) for Spark, as well as Spark SQL, Presto, and Apache Drill. Performance when running such tasks is highly sub-optimal compared to specialized systems, but relatively high latency makes it unrealistic to use these systems as the backend for a web interface.
-
-## What If I Have a Problem with Encodings When Using Oracle Through ODBC? {#oracle-odbc-encodings}
-
-If you use Oracle through the ODBC driver as a source of external dictionaries, you need to set the correct value for the `NLS_LANG` environment variable in `/etc/default/clickhouse`. For more information, see the [Oracle NLS\_LANG FAQ](https://www.oracle.com/technetwork/products/globalization/nls-lang-099431.html).
-
-**Example**
-
-``` sql
-NLS_LANG=RUSSIAN_RUSSIA.UTF8
-```
-
-## How Do I Export Data from ClickHouse to a File? {#how-to-export-to-file}
-
-### Using INTO OUTFILE Clause {#using-into-outfile-clause}
-
-Add an [INTO OUTFILE](../sql-reference/statements/select/into-outfile.md#into-outfile-clause) clause to your query.
-
-For example:
-
-``` sql
-SELECT * FROM table INTO OUTFILE 'file'
-```
-
-By default, ClickHouse uses the [TabSeparated](../interfaces/formats.md#tabseparated) format for output data. To select the [data format](../interfaces/formats.md), use the [FORMAT clause](../sql-reference/statements/select/format.md#format-clause).
-
-For example:
-
-``` sql
-SELECT * FROM table INTO OUTFILE 'file' FORMAT CSV
-```
-
-### Using a File-Engine Table {#using-a-file-engine-table}
-
-See [File](../engines/table-engines/special/file.md).
-
-### Using Command-Line Redirection {#using-command-line-redirection}
-
-``` sql
-$ clickhouse-client --query "SELECT * from table" --format FormatName > result.txt
-```
-
-See [clickhouse-client](../interfaces/cli.md).
-
-{## [Original article](https://clickhouse.tech/docs/en/faq/general/) ##}

docs/en/faq/general/dbms-naming.md

@@ -0,0 +1,11 @@
+---
+toc_hidden: true
+toc_priority: 10
+---
+
+# What Does “ClickHouse” Mean? {#what-does-clickhouse-mean}
+
+It’s a combination of “**Click**stream” and “Data ware**house**”. It comes from the original use case at Yandex.Metrica, where ClickHouse was supposed to keep records of all clicks by people from all over the Internet and it still does the job. You can read more about this use case on [ClickHouse history](../../introduction/history.md) page.
+
+!!! info "Fun fact"
+    Many years after ClickHouse got its name, this approach of combining two words that are meaningful on their own has been highlighted as the best way to name a database in a [research by Andy Pavlo](https://www.cs.cmu.edu/~pavlo/blog/2020/03/on-naming-a-database-management-system.html), an Associate Professor of Databases at Carnegie Mellon University. ClickHouse shared his “best database name of all time” award with Postgres.

docs/en/faq/general/index.md

@@ -0,0 +1,18 @@
+---
+toc_hidden_folder: true
+toc_priority: 1
+toc_title: General
+---
+
+# General Questions About ClickHouse {#general-questions}
+
+Questions:
+
+-   [What does “ClickHouse” mean?](../../faq/general/dbms-naming.md)
+-   [What does “Не тормозит” mean?](../../faq/general/ne-tormozit.md)
+-   [Why not use something like MapReduce?](../../faq/general/mapreduce.md)
+
+!!! info "Don’t see what you were looking for?"
+    Check out [other F.A.Q. categories](../../faq/index.md) or browse around main documentation articles found in the left sidebar.
+
+{## [Original article](https://clickhouse.tech/docs/en/faq/general/) ##}

docs/en/faq/general/mapreduce.md

@@ -0,0 +1,12 @@
+---
+toc_hidden: true
+toc_priority: 20
+---
+
+# Why Not Use Something Like MapReduce? {#why-not-use-something-like-mapreduce}
+
+We can refer to systems like MapReduce as distributed computing systems in which the reduce operation is based on distributed sorting. The most common open-source solution in this class is [Apache Hadoop](http://hadoop.apache.org). Yandex uses its in-house solution, YT.
+
+These systems aren’t appropriate for online queries due to their high latency. In other words, they can’t be used as the back-end for a web interface. These types of systems aren’t useful for real-time data updates. Distributed sorting isn’t the best way to perform reduce operations if the result of the operation and all the intermediate results (if there are any) are located in the RAM of a single server, which is usually the case for online queries. In such a case, a hash table is an optimal way to perform reduce operations. A common approach to optimizing map-reduce tasks is pre-aggregation (partial reduce) using a hash table in RAM. The user performs this optimization manually. Distributed sorting is one of the main causes of reduced performance when running simple map-reduce tasks.
+
+Most MapReduce implementations allow you to execute arbitrary code on a cluster. But a declarative query language is better suited to OLAP to run experiments quickly. For example, Hadoop has Hive and Pig. Also consider Cloudera Impala or Shark (outdated) for Spark, as well as Spark SQL, Presto, and Apache Drill. Performance when running such tasks is highly sub-optimal compared to specialized systems, but relatively high latency makes it unrealistic to use these systems as the backend for a web interface.

docs/en/faq/general/ne-tormozit.md

@@ -0,0 +1,24 @@
+---
+toc_hidden: true
+toc_priority: 11
+---
+
+# What Does “Не тормозит” mean? {#what-does-ne-tormozit-mean}
+
+This question usually arises when people see official ClickHouse t-shirts. They have large words **“ClickHouse не тормозит”** on the front.
+
+Before ClickHouse became open-source, it has been developed as an in-house storage system by the largest Russian IT company, [Yandex](https://yandex.com/company/). That’s why it initially got its slogan in Russian, which is “не тормозит”. After the open-source release we first produced some of those t-shirts for events in Russia and it was a no-brainer to use the slogan as-is.
+
+One of the following batches of those t-shirts was supposed to be given away on events outside of Russia and we tried to make the English version of the slogan. Unfortunately, the Russian language is kind of elegant in terms of expressing stuff and there was a restriction of limited space on a t-shirt, so we failed to come up with good enough translation (most options appeared to be either long or inaccurate) and decided to keep the slogan in Russian even on t-shirts produced for international events. It appeared to be a great decision because people all over the world get positively surprised and curious when they see it.
+
+So, what does it mean? Here are some ways to translate *“не тормозит”*:
+
+-   If you translate it literally, it’d be something like *“ClickHouse doesn’t press the brake pedal”*.
+-   If you’d want to express it as close to how it sounds to a Russian person with IT background, it’d be something like *“If you larger system lags, it’s not because it uses ClickHouse”*.
+-   Shorter, but not so precise versions could be *“ClickHouse is not slow”*, *“ClickHouse doesn’t lag”* or just *“ClickHouse is fast”*.
+
+If you haven’t seen one of those t-shirts in person, you can check them out online in many ClickHouse-related videos. For example, this one:
+
+![iframe](https://www.youtube.com/embed/bSyQahMVZ7w)
+
+P.S. These t-shirts are not for sale, they are given away for free on most [ClickHouse Meetups](https://clickhouse.tech/#meet), usually for best questions or other forms of active participation.

docs/en/faq/index.md

@@ -2,7 +2,16 @@
 toc_folder_title: F.A.Q.
 toc_hidden: true
 toc_priority: 76
-toc_title: hidden
 ---
 
+# ClickHouse F.A.Q {#clickhouse-f-a-q}
+
+This section of the documentation is a place to collect answers to ClickHouse-related questions that arise often.
+
+Categories:
+
+-   [General](../faq/general/index.md)
+-   [Operations](../faq/operations/index.md)
+-   [Integration](../faq/integration/index.md)
+
 {## [Original article](https://clickhouse.tech/docs/en/faq) ##}

docs/en/faq/integration/file-export.md

@@ -0,0 +1,36 @@
+---
+toc_hidden: true
+toc_priority: 10
+---
+
+# How Do I Export Data from ClickHouse to a File? {#how-to-export-to-file}
+
+## Using INTO OUTFILE Clause {#using-into-outfile-clause}
+
+Add an [INTO OUTFILE](../../sql-reference/statements/select/into-outfile.md#into-outfile-clause) clause to your query.
+
+For example:
+
+``` sql
+SELECT * FROM table INTO OUTFILE 'file'
+```
+
+By default, ClickHouse uses the [TabSeparated](../../interfaces/formats.md#tabseparated) format for output data. To select the [data format](../../interfaces/formats.md), use the [FORMAT clause](../../sql-reference/statements/select/format.md#format-clause).
+
+For example:
+
+``` sql
+SELECT * FROM table INTO OUTFILE 'file' FORMAT CSV
+```
+
+## Using a File-Engine Table {#using-a-file-engine-table}
+
+See [File](../../engines/table-engines/special/file.md) table engine.
+
+## Using Command-Line Redirection {#using-command-line-redirection}
+
+``` sql
+$ clickhouse-client --query "SELECT * from table" --format FormatName > result.txt
+```
+
+See [clickhouse-client](../../interfaces/cli.md).

docs/en/faq/integration/index.md

@@ -0,0 +1,17 @@
+---
+toc_hidden_folder: true
+toc_priority: 3
+toc_title: Integration
+---
+
+# Question About Integrating ClickHouse and Other Systems {#question-about-integrating-clickhouse-and-other-systems}
+
+Questions:
+
+-   [How do I export data from ClickHouse to a file?](../../faq/integration/file-export.md)
+-   [What if I Have a problem with encodings when connecting to Oracle via ODBC?](../../faq/integration/oracle-odbc.md)
+
+!!! info "Don’t see what you were looking for?"
+    Check out [other F.A.Q. categories](../../faq/index.md) or browse around main documentation articles found in the left sidebar.
+
+{## [Original article](https://clickhouse.tech/docs/en/faq/integration/) ##}

docs/en/faq/integration/oracle-odbc.md

@@ -0,0 +1,14 @@
+---
+toc_hidden: true
+toc_priority: 20
+---
+
+# What If I Have a Problem with Encodings When Using Oracle Via ODBC? {#oracle-odbc-encodings}
+
+If you use Oracle as a source of ClickHouse external dictionaries via Oracle ODBC driver, you need to set the correct value for the `NLS_LANG` environment variable in `/etc/default/clickhouse`. For more information, see the [Oracle NLS\_LANG FAQ](https://www.oracle.com/technetwork/products/globalization/nls-lang-099431.html).
+
+**Example**
+
+``` sql
+NLS_LANG=RUSSIAN_RUSSIA.UTF8
+```

docs/en/faq/operations/index.md

@@ -0,0 +1,16 @@
+---
+toc_hidden_folder: true
+toc_priority: 2
+toc_title: Operations
+---
+
+# Question About Operating ClickHouse Servers and Clusters {#question-about-operating-clickhouse-servers-and-clusters}
+
+Questions:
+
+-   [Which ClickHouse version to use in production?](../../faq/operations/production.md)
+
+!!! info "Don’t see what you were looking for?"
+    Check out [other F.A.Q. categories](../../faq/index.md) or browse around main documentation articles found in the left sidebar.
+
+{## [Original article](https://clickhouse.tech/docs/en/faq/production/) ##}

docs/en/faq/operations/production.md

@@ -0,0 +1,69 @@
+---
+toc_hidden: true
+toc_priority: 10
+---
+
+# Which ClickHouse Version to Use in Production? {#which-clickhouse-version-to-use-in-production}
+
+First of all, let’s discuss why people ask this question in the first place. There are two key reasons:
+
+1.  ClickHouse is developed with pretty high velocity and usually, there are 10+ stable releases per year. It makes a wide range of releases to choose from, which is not so trivial choice.
+2.  Some users want to avoid spending time figuring out which version works best for their use case and just follow someone else’s advice.
+
+The second reason is more fundamental, so we’ll start with it and then get back to navigating through various ClickHouse releases.
+
+## Which ClickHouse Version Do You Recommend? {#which-clickhouse-version-do-you-recommend}
+
+It’s tempting to hire consultants or trust some known experts to get rid of responsibility for your production environment. You install some specific ClickHouse version that someone else recommended, now if there’s some issue with it - it’s not your fault, it’s someone else’s. This line of reasoning is a big trap. No external person knows better what’s going on in your company’s production environment.
+
+So how to properly choose which ClickHouse version to upgrade to? Or how to choose your first ClickHouse version? First of all, you need to invest in setting up a **realistic pre-production environment**. In an ideal world, it could be a completely identical shadow copy, but that’s usually expensive.
+
+Here’re some key points to get reasonable fidelity in a pre-production environment with not so high costs:
+
+-   Pre-production environment needs to run an as close set of queries as you intend to run in production:
+    -   Don’t make it read-only with some frozen data.
+    -   Don’t make it write-only with just copying data without building some typical reports.
+    -   Don’t wipe it clean instead of applying schema migrations.
+-   Use a sample of real production data and queries. Try to choose a sample that’s still representative and makes `SELECT` queries return reasonable results. Use obfuscation if your data is sensitive and internal policies don’t allow it to leave the production environment.
+-   Make sure that pre-production is covered by your monitoring and alerting software the same way as your production environment does.
+-   If your production spans across multiple datacenters or regions, make your pre-production does the same.
+-   If your production uses complex features like replication, distributed table, cascading materialize views, make sure they are configured similarly in pre-production.
+-   There’s a trade-off on using the roughly same number of servers or VMs in pre-production as in production, but of smaller size, or much less of them, but of the same size. The first option might catch extra network-related issues, while the latter is easier to manage.
+
+The second area to invest in is **automated testing infrastructure**. Don’t assume that if some kind of query has executed successfully once, it’ll continue to do so forever. It’s ok to have some unit tests where ClickHouse is mocked but make sure your product has a reasonable set of automated tests that are run against real ClickHouse and check that all important use cases are still working as expected.
+
+Extra step forward could be contributing those automated tests to [ClickHouse’s open-source test infrastructure](https://github.com/ClickHouse/ClickHouse/tree/master/tests) that’s continuously used in its day-to-day development. It definitely will take some additional time and effort to learn [how to run it](../../development/tests.md) and then how to adapt your tests to this framework, but it’ll pay off by ensuring that ClickHouse releases are already tested against them when they are announced stable, instead of repeatedly losing time on reporting the issue after the fact and then waiting for a bugfix to be implemented, backported and released. Some companies even have such test contributions to infrastructure by its use as an internal policy, most notably it’s called [Beyonce’s Rule](https://www.oreilly.com/library/view/software-engineering-at/9781492082781/ch01.html#policies_that_scale_well) at Google.
+
+When you have your pre-production environment and testing infrastructure in place, choosing the best version is straightforward:
+
+1.  Routinely run your automated tests against new ClickHouse releases. You can do it even for ClickHouse releases that are marked as `testing`, but going forward to the next steps with them is not recommended.
+2.  Deploy the ClickHouse release that passed the tests to pre-production and check that all processes are running as expected.
+3.  Report any issues you discovered to [ClickHouse GitHub Issues](https://github.com/ClickHouse/ClickHouse/issues).
+4.  If there were no major issues, it should be safe to start deploying ClickHouse release to your production environment. Investing in gradual release automation that implements an approach similar to [canary releases](https://martinfowler.com/bliki/CanaryRelease.html) or [green-blue deployments](https://martinfowler.com/bliki/BlueGreenDeployment.html) might further reduce the risk of issues in production.
+
+As you might have noticed, there’s nothing specific to ClickHouse in the approach described above, people do that for any piece of infrastructure they rely on if they take their production environment seriously.
+
+## How to Choose Between ClickHouse Releases? {#how-to-choose-between-clickhouse-releases}
+
+If you look into contents of ClickHouse package repository, you’ll see four kinds of packages:
+
+1.  `testing`
+2.  `prestable`
+3.  `stable`
+4.  `lts` (long-term support)
+
+As was mentioned earlier, `testing` is good mostly to notice issues early, running them in production is not recommended because each of them is not tested as thoroughly as other kinds of packages.
+
+`prestable` is a release candidate which generally looks promising and is likely to become announced as `stable` soon. You can try them out in pre-production and report issues if you see any.
+
+For production use, there are two key options: `stable` and `lts`. Here is some guidance on how to choose between them:
+
+-   `stable` is the kind of package we recommend by default. They are released roughly monthly (and thus provide new features with reasonable delay) and three latest stable releases are supported in terms of diagnostics and backporting of bugfixes.
+-   `lts` are released twice a year and are supported for a year after their initial release. You might prefer them over `stable` in the following cases:
+    -   Your company has some internal policies that don’t allow for frequent upgrades or using non-LTS software.
+    -   You are using ClickHouse in some secondary products that either doesn’t require any complex ClickHouse features and don’t have enough resources to keep it updated.
+
+Many teams who initially thought that `lts` is the way to go, often switch to `stable` anyway because of some recent feature that’s important for their product.
+
+!!! warning "Important"
+    One more thing to keep in mind when upgrading ClickHouse: we’re always keeping eye on compatibility across releases, but sometimes it’s not reasonable to keep and some minor details might change. So make sure you check the [changelog](../../whats-new/changelog/index.md) before upgrading to see if there are any notes about backward-incompatible changes.

docs/en/sql-reference/dictionaries/external-dictionaries/external-dicts-dict-sources.md

@@ -206,7 +206,7 @@ Setting fields:
 
 ClickHouse receives quoting symbols from ODBC-driver and quote all settings in queries to driver, so it’s necessary to set table name accordingly to table name case in database.
 
-If you have a problems with encodings when using Oracle, see the corresponding [FAQ](../../../faq/general.md#oracle-odbc-encodings) article.
+If you have a problems with encodings when using Oracle, see the corresponding [F.A.Q.](../../../faq/integration/oracle-odbc.md) item.
 
 ### Known Vulnerability of the ODBC Dictionary Functionality {#known-vulnerability-of-the-odbc-dictionary-functionality}
 

docs/tools/amp.py

@@ -85,6 +85,15 @@ def html_to_amp(content):
                 tag.attrs['width'] = '640'
             if not tag.attrs.get('height'):
                 tag.attrs['height'] = '320'
+        if tag.name == 'iframe':
+            tag.name = 'amp-iframe'
+            tag.attrs['layout'] = 'responsive'
+            del tag.attrs['alt']
+            del tag.attrs['allowfullscreen']
+            if not tag.attrs.get('width'):
+                tag.attrs['width'] = '640'
+            if not tag.attrs.get('height'):
+                tag.attrs['height'] = '320'
         elif tag.name == 'a':
             href = tag.attrs.get('href')
             if href:

docs/tools/nav.py

@@ -1,5 +1,6 @@
 import collections
 import datetime
+import hashlib
 import logging
 import os
 
@@ -39,13 +40,17 @@ def build_nav_entry(root, args):
                 title = meta.get('toc_folder_title', 'hidden')
             prio = meta.get('toc_priority', 9999)
             logging.debug(f'Nav entry: {prio}, {title}, {path}')
-            if not content.strip():
+            if meta.get('toc_hidden') or not content.strip():
                 title = 'hidden'
+            if title == 'hidden':
+                title = 'hidden-' + hashlib.sha1(content.encode('utf-8')).hexdigest()
             if args.nav_limit and len(result_items) >= args.nav_limit:
                 break
             result_items.append((prio, title, path))
     result_items = sorted(result_items, key=lambda x: (x[0], x[1]))
     result = collections.OrderedDict([(item[1], item[2]) for item in result_items])
+    if index_meta.get('toc_hidden_folder'):
+        current_title += '|hidden-folder'
     return index_meta.get('toc_priority', 10000), current_title, result
 
 

website/templates/docs/amp.html

@@ -13,6 +13,7 @@
     <style amp-boilerplate>body{-webkit-animation:-amp-start 8s steps(1,end) 0s 1 normal both;-moz-animation:-amp-start 8s steps(1,end) 0s 1 normal both;-ms-animation:-amp-start 8s steps(1,end) 0s 1 normal both;animation:-amp-start 8s steps(1,end) 0s 1 normal both}@-webkit-keyframes -amp-start{from{visibility:hidden}to{visibility:visible}}@-moz-keyframes -amp-start{from{visibility:hidden}to{visibility:visible}}@-ms-keyframes -amp-start{from{visibility:hidden}to{visibility:visible}}@-o-keyframes -amp-start{from{visibility:hidden}to{visibility:visible}}@keyframes -amp-start{from{visibility:hidden}to{visibility:visible}}</style><noscript><style amp-boilerplate>body{-webkit-animation:none;-moz-animation:none;-ms-animation:none;animation:none}</style></noscript>
     <style amp-custom>CUSTOM_CSS_PLACEHOLDER</style>
     <script async custom-element="amp-analytics" src="https://cdn.ampproject.org/v0/amp-analytics-0.1.js"></script>
+    <script async custom-element="amp-iframe" src="https://cdn.ampproject.org/v0/amp-iframe-0.1.js"></script>
   </head>
   <body class="amp" dir="{{ direction or 'ltr' }}">
     <div class="container-fluid container-lg pb-5">

website/templates/docs/content.html

@@ -11,7 +11,11 @@
                     <nav class="col text-muted" aria-label="breadcrumb">
                         <ol class="breadcrumb bg-transparent p-0">
                             {% for ancestor in ancestors %}
-                            <li class="breadcrumb-item"><a href="{{ '../' * ancestor_ns.level }}" class="text-reset">{{ ancestor.title.strip() }}</a>
+                            {% set ancestor_title = ancestor.title.strip() %}
+                            {% if ancestor_title.endswith('|hidden-folder') %}
+                                {% set ancestor_title = ancestor_title[:-14] %}
+                            {% endif %}
+                            <li class="breadcrumb-item"><a href="{{ '../' * ancestor_ns.level }}" class="text-reset">{{ ancestor_title }}</a>
                                 {% set ancestor_ns.level = ancestor_ns.level - 1 %}
                             </li>
                            {% endfor %}

website/templates/docs/nav.html

@@ -6,7 +6,7 @@
         <div class="w-100 navbar-text text-left d-none d-md-block">
             <h1 class="h3 m-0 p-0 d-inline">
                 <a href="#" class="text-white text-decoration-none">
-                    {% if page.title != 'hidden' %}
+                    {% if not page.title.startswith('hidden') %}
                         {{ page.title or config.site_name }}
                     {% else %}
                         {{ page.meta.toc_folder_title or config.site_name }}

website/templates/docs/sidebar-item.html

@@ -1,6 +1,8 @@
 {% set collapsed_threshold = 1 %}
-{% if nav_item.title != "hidden" %}
-{% if nav_item.children %}
+{% if not nav_item.title.startswith("hidden-") %}
+{% if nav_item.title.endswith('|hidden-folder') %}
+<a href="{{ base_url }}/{{ nav_item.children[0].url }}" title="{{ nav_item.title[:-14] }}" class="nav-link{% if nav_item.active%} text-white active{% else %} text-light{% endif %} text-wrap d-inline-block w-100 px-0">{{ nav_item.title[:-14] }}</a>
+{% elif nav_item.children %}
 <a href="#sidebar-{{ path }}" data-toggle="collapse" aria-expanded="{% if level < collapsed_threshold %}true{% else %}false{% endif %}" aria-controls="sidebar-{{ path }}" title="{{ nav_item.title }}" class="nav-link text-light text-wrap dropdown-toggle px-0">{{ nav_item.title }}</a>
 <div id="sidebar-{{ path }}" class="nav flex-column collapse{% if level < collapsed_threshold %} show{% endif %}">
     <nav class="nav flex-column pl-2">