thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-10 01:25:21 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update external_dicts_dict_sources.md

Browse Source
This commit is contained in:
alexey-milovidov 2018-11-13 20:52:36 +03:00 committed by GitHub
parent 062a56b9fc
commit 55a576f762
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 432 additions and 432 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

864
docs/en/query_language/dicts/external_dicts_dict_sources.md
View File

@ -1,433 +1,433 @@
<a name="dicts-external_dicts_dict_sources"></a> <a name="dicts-external_dicts_dict_sources"></a>
# Sources of External Dictionaries # Sources of External Dictionaries
An external dictionary can be connected from many different sources. An external dictionary can be connected from many different sources.
The configuration looks like this: The configuration looks like this:
```xml ```xml
<yandex> <yandex>
<dictionary> <dictionary>
... ...
<source> <source>
<source_type> <source_type>
<!-- Source configuration --> <!-- Source configuration -->
</source_type> </source_type>
</source> </source>
... ...
</dictionary> </dictionary>
... ...
</yandex> </yandex>
``` ```
The source is configured in the `source` section. The source is configured in the `source` section.
Types of sources (`source_type`): Types of sources (`source_type`):
- [Local file](#dicts-external_dicts_dict_sources-local_file) - [Local file](#dicts-external_dicts_dict_sources-local_file)
- [Executable file](#dicts-external_dicts_dict_sources-executable) - [Executable file](#dicts-external_dicts_dict_sources-executable)
- [HTTP(s)](#dicts-external_dicts_dict_sources-http) - [HTTP(s)](#dicts-external_dicts_dict_sources-http)
- [ODBC](#dicts-external_dicts_dict_sources-odbc) - DBMS
- DBMS - [MySQL](#dicts-external_dicts_dict_sources-mysql)
- [MySQL](#dicts-external_dicts_dict_sources-mysql) - [ClickHouse](#dicts-external_dicts_dict_sources-clickhouse)
- [ClickHouse](#dicts-external_dicts_dict_sources-clickhouse) - [MongoDB](#dicts-external_dicts_dict_sources-mongodb)
- [MongoDB](#dicts-external_dicts_dict_sources-mongodb) - [ODBC](#dicts-external_dicts_dict_sources-odbc)
<a name="dicts-external_dicts_dict_sources-local_file"></a> <a name="dicts-external_dicts_dict_sources-local_file"></a>
## Local File ## Local File
Example of settings: Example of settings:
```xml ```xml
<source> <source>
<file> <file>
<path>/opt/dictionaries/os.tsv</path> <path>/opt/dictionaries/os.tsv</path>
<format>TabSeparated</format> <format>TabSeparated</format>
</file> </file>
</source> </source>
``` ```
Setting fields: Setting fields:
- `path` – The absolute path to the file. - `path` – The absolute path to the file.
- `format` – The file format. All the formats described in "[Formats](../../interfaces/formats.md#formats)" are supported. - `format` – The file format. All the formats described in "[Formats](../../interfaces/formats.md#formats)" are supported.
<a name="dicts-external_dicts_dict_sources-executable"></a> <a name="dicts-external_dicts_dict_sources-executable"></a>
## Executable File ## Executable File
Working with executable files depends on [how the dictionary is stored in memory](external_dicts_dict_layout.md#dicts-external_dicts_dict_layout). If the dictionary is stored using `cache` and `complex_key_cache`, ClickHouse requests the necessary keys by sending a request to the executable file's `STDIN`. Working with executable files depends on [how the dictionary is stored in memory](external_dicts_dict_layout.md#dicts-external_dicts_dict_layout). If the dictionary is stored using `cache` and `complex_key_cache`, ClickHouse requests the necessary keys by sending a request to the executable file's `STDIN`.
Example of settings: Example of settings:
```xml ```xml
<source> <source>
<executable> <executable>
<command>cat /opt/dictionaries/os.tsv</command> <command>cat /opt/dictionaries/os.tsv</command>
<format>TabSeparated</format> <format>TabSeparated</format>
</executable> </executable>
</source> </source>
``` ```
Setting fields: Setting fields:
- `command` – The absolute path to the executable file, or the file name (if the program directory is written to `PATH`). - `command` – The absolute path to the executable file, or the file name (if the program directory is written to `PATH`).
- `format` – The file format. All the formats described in "[Formats](../../interfaces/formats.md#formats)" are supported. - `format` – The file format. All the formats described in "[Formats](../../interfaces/formats.md#formats)" are supported.
<a name="dicts-external_dicts_dict_sources-http"></a> <a name="dicts-external_dicts_dict_sources-http"></a>
## HTTP(s) ## HTTP(s)
Working with an HTTP(s) server depends on [how the dictionary is stored in memory](external_dicts_dict_layout.md#dicts-external_dicts_dict_layout). If the dictionary is stored using `cache` and `complex_key_cache`, ClickHouse requests the necessary keys by sending a request via the `POST` method. Working with an HTTP(s) server depends on [how the dictionary is stored in memory](external_dicts_dict_layout.md#dicts-external_dicts_dict_layout). If the dictionary is stored using `cache` and `complex_key_cache`, ClickHouse requests the necessary keys by sending a request via the `POST` method.
Example of settings: Example of settings:
```xml ```xml
<source> <source>
<http> <http>
<url>http://[::1]/os.tsv</url> <url>http://[::1]/os.tsv</url>
<format>TabSeparated</format> <format>TabSeparated</format>
</http> </http>
</source> </source>
``` ```
In order for ClickHouse to access an HTTPS resource, you must [configure openSSL](../../operations/server_settings/settings.md#server_settings-openSSL) in the server configuration. In order for ClickHouse to access an HTTPS resource, you must [configure openSSL](../../operations/server_settings/settings.md#server_settings-openSSL) in the server configuration.
Setting fields: Setting fields:
- `url` – The source URL. - `url` – The source URL.
- `format` – The file format. All the formats described in "[Formats](../../interfaces/formats.md#formats)" are supported. - `format` – The file format. All the formats described in "[Formats](../../interfaces/formats.md#formats)" are supported.
<a name="dicts-external_dicts_dict_sources-odbc"></a> <a name="dicts-external_dicts_dict_sources-odbc"></a>
## ODBC ## ODBC
You can use this method to connect any database that has an ODBC driver. You can use this method to connect any database that has an ODBC driver.
Example of settings: Example of settings:
```xml ```xml
<odbc> <odbc>
<db>DatabaseName</db> <db>DatabaseName</db>
<table>ShemaName.TableName</table> <table>ShemaName.TableName</table>
<connection_string>DSN=some_parameters</connection_string> <connection_string>DSN=some_parameters</connection_string>
<invalidate_query>SQL_QUERY</invalidate_query> <invalidate_query>SQL_QUERY</invalidate_query>
</odbc> </odbc>
``` ```
Setting fields: Setting fields:
- `db` – Name of the database. Omit it if the database name is set in the `<connection_string>` parameters. - `db` – Name of the database. Omit it if the database name is set in the `<connection_string>` parameters.
- `table` – Name of the table and schema if exists. - `table` – Name of the table and schema if exists.
- `connection_string` – Connection string. - `connection_string` – Connection string.
- `invalidate_query` – Query for checking the dictionary status. Optional parameter. Read more in the section [Updating dictionaries](external_dicts_dict_lifetime.md#dicts-external_dicts_dict_lifetime). - `invalidate_query` – Query for checking the dictionary status. Optional parameter. Read more in the section [Updating dictionaries](external_dicts_dict_lifetime.md#dicts-external_dicts_dict_lifetime).
ClickHouse receives quoting symbols from ODBC-driver and quote all settings in queries to driver, so it's necessary to set table name accordingly to table name case in database. ClickHouse receives quoting symbols from ODBC-driver and quote all settings in queries to driver, so it's necessary to set table name accordingly to table name case in database.
### Known vulnerability of the ODBC dictionary functionality ### Known vulnerability of the ODBC dictionary functionality
!!! attention !!! attention
When connecting to the database through the ODBC driver connection parameter `Servername` can be substituted. In this case values of `USERNAME` and `PASSWORD` from `odbc.ini` are sent to the remote server and can be compromised. When connecting to the database through the ODBC driver connection parameter `Servername` can be substituted. In this case values of `USERNAME` and `PASSWORD` from `odbc.ini` are sent to the remote server and can be compromised.
**Example of insecure use** **Example of insecure use**
Let's configure unixODBC for PostgreSQL. Content of `/etc/odbc.ini`: Let's configure unixODBC for PostgreSQL. Content of `/etc/odbc.ini`:
``` ```
[gregtest] [gregtest]
Driver = /usr/lib/psqlodbca.so Driver = /usr/lib/psqlodbca.so
Servername = localhost Servername = localhost
PORT = 5432 PORT = 5432
DATABASE = test_db DATABASE = test_db
#OPTION = 3 #OPTION = 3
USERNAME = test USERNAME = test
PASSWORD = test PASSWORD = test
``` ```
If you then make a query such as If you then make a query such as
``` ```
SELECT * FROM odbc('DSN=gregtest;Servername=some-server.com', 'test_db'); SELECT * FROM odbc('DSN=gregtest;Servername=some-server.com', 'test_db');
``` ```
ODBC driver will send values of `USERNAME` and `PASSWORD` from `odbc.ini` to `some-server.com`. ODBC driver will send values of `USERNAME` and `PASSWORD` from `odbc.ini` to `some-server.com`.
### Example of Connecting PostgreSQL ### Example of Connecting PostgreSQL
Ubuntu OS. Ubuntu OS.
Installing unixODBC and the ODBC driver for PostgreSQL: Installing unixODBC and the ODBC driver for PostgreSQL:
sudo apt-get install -y unixodbc odbcinst odbc-postgresql sudo apt-get install -y unixodbc odbcinst odbc-postgresql
Configuring `/etc/odbc.ini` (or `~/.odbc.ini`): Configuring `/etc/odbc.ini` (or `~/.odbc.ini`):
``` ```
[DEFAULT] [DEFAULT]
Driver = myconnection Driver = myconnection
[myconnection] [myconnection]
Description = PostgreSQL connection to my_db Description = PostgreSQL connection to my_db
Driver = PostgreSQL Unicode Driver = PostgreSQL Unicode
Database = my_db Database = my_db
Servername = 127.0.0.1 Servername = 127.0.0.1
UserName = username UserName = username
Password = password Password = password
Port = 5432 Port = 5432
Protocol = 9.3 Protocol = 9.3
ReadOnly = No ReadOnly = No
RowVersioning = No RowVersioning = No
ShowSystemTables = No ShowSystemTables = No
ConnSettings = ConnSettings =
``` ```
The dictionary configuration in ClickHouse: The dictionary configuration in ClickHouse:
```xml ```xml
<yandex> <yandex>
<dictionary> <dictionary>
<name>table_name</name> <name>table_name</name>
<source> <source>
<odbc> <odbc>
<!-- You can specify the following parameters in connection_string: --> <!-- You can specify the following parameters in connection_string: -->
<!-- DSN=myconnection;UID=username;PWD=password;HOST=127.0.0.1;PORT=5432;DATABASE=my_db --> <!-- DSN=myconnection;UID=username;PWD=password;HOST=127.0.0.1;PORT=5432;DATABASE=my_db -->
<connection_string>DSN=myconnection</connection_string> <connection_string>DSN=myconnection</connection_string>
<table>postgresql_table</table> <table>postgresql_table</table>
</odbc> </odbc>
</source> </source>
<lifetime> <lifetime>
<min>300</min> <min>300</min>
<max>360</max> <max>360</max>
</lifetime> </lifetime>
<layout> <layout>
<hashed/> <hashed/>
</layout> </layout>
<structure> <structure>
<id> <id>
<name>id</name> <name>id</name>
</id> </id>
<attribute> <attribute>
<name>some_column</name> <name>some_column</name>
<type>UInt64</type> <type>UInt64</type>
<null_value>0</null_value> <null_value>0</null_value>
</attribute> </attribute>
</structure> </structure>
</dictionary> </dictionary>
</yandex> </yandex>
``` ```
You may need to edit `odbc.ini` to specify the full path to the library with the driver `DRIVER=/usr/local/lib/psqlodbcw.so`. You may need to edit `odbc.ini` to specify the full path to the library with the driver `DRIVER=/usr/local/lib/psqlodbcw.so`.
### Example of Connecting MS SQL Server ### Example of Connecting MS SQL Server
Ubuntu OS. Ubuntu OS.
Installing the driver: : Installing the driver: :
``` ```
sudo apt-get install tdsodbc freetds-bin sqsh sudo apt-get install tdsodbc freetds-bin sqsh
``` ```
Configuring the driver: : Configuring the driver: :
``` ```
$ cat /etc/freetds/freetds.conf $ cat /etc/freetds/freetds.conf
... ...
[MSSQL] [MSSQL]
host = 192.168.56.101 host = 192.168.56.101
port = 1433 port = 1433
tds version = 7.0 tds version = 7.0
client charset = UTF-8 client charset = UTF-8
$ cat /etc/odbcinst.ini $ cat /etc/odbcinst.ini
... ...
[FreeTDS] [FreeTDS]
Description = FreeTDS Description = FreeTDS
Driver = /usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so Driver = /usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so
Setup = /usr/lib/x86_64-linux-gnu/odbc/libtdsS.so Setup = /usr/lib/x86_64-linux-gnu/odbc/libtdsS.so
FileUsage = 1 FileUsage = 1
UsageCount = 5 UsageCount = 5
$ cat ~/.odbc.ini $ cat ~/.odbc.ini
... ...
[MSSQL] [MSSQL]
Description = FreeTDS Description = FreeTDS
Driver = FreeTDS Driver = FreeTDS
Servername = MSSQL Servername = MSSQL
Database = test Database = test
UID = test UID = test
PWD = test PWD = test
Port = 1433 Port = 1433
``` ```
Configuring the dictionary in ClickHouse: Configuring the dictionary in ClickHouse:
```xml ```xml
<yandex> <yandex>
<dictionary> <dictionary>
<name>test</name> <name>test</name>
<source> <source>
<odbc> <odbc>
<table>dict</table> <table>dict</table>
<connection_string>DSN=MSSQL;UID=test;PWD=test</connection_string> <connection_string>DSN=MSSQL;UID=test;PWD=test</connection_string>
</odbc> </odbc>
</source> </source>
<lifetime> <lifetime>
<min>300</min> <min>300</min>
<max>360</max> <max>360</max>
</lifetime> </lifetime>
<layout> <layout>
<flat /> <flat />
</layout> </layout>
<structure> <structure>
<id> <id>
<name>k</name> <name>k</name>
</id> </id>
<attribute> <attribute>
<name>s</name> <name>s</name>
<type>String</type> <type>String</type>
<null_value></null_value> <null_value></null_value>
</attribute> </attribute>
</structure> </structure>
</dictionary> </dictionary>
</yandex> </yandex>
``` ```
## DBMS ## DBMS
<a name="dicts-external_dicts_dict_sources-mysql"></a> <a name="dicts-external_dicts_dict_sources-mysql"></a>
### MySQL ### MySQL
Example of settings: Example of settings:
```xml ```xml
<source> <source>
<mysql> <mysql>
<port>3306</port> <port>3306</port>
<user>clickhouse</user> <user>clickhouse</user>
<password>qwerty</password> <password>qwerty</password>
<replica> <replica>
<host>example01-1</host> <host>example01-1</host>
<priority>1</priority> <priority>1</priority>
</replica> </replica>
<replica> <replica>
<host>example01-2</host> <host>example01-2</host>
<priority>1</priority> <priority>1</priority>
</replica> </replica>
<db>db_name</db> <db>db_name</db>
<table>table_name</table> <table>table_name</table>
<where>id=10</where> <where>id=10</where>
<invalidate_query>SQL_QUERY</invalidate_query> <invalidate_query>SQL_QUERY</invalidate_query>
</mysql> </mysql>
</source> </source>
``` ```
Setting fields: Setting fields:
- `port` – The port on the MySQL server. You can specify it for all replicas, or for each one individually (inside `<replica>`). - `port` – The port on the MySQL server. You can specify it for all replicas, or for each one individually (inside `<replica>`).
- `user` – Name of the MySQL user. You can specify it for all replicas, or for each one individually (inside `<replica>`). - `user` – Name of the MySQL user. You can specify it for all replicas, or for each one individually (inside `<replica>`).
- `password` – Password of the MySQL user. You can specify it for all replicas, or for each one individually (inside `<replica>`). - `password` – Password of the MySQL user. You can specify it for all replicas, or for each one individually (inside `<replica>`).
- `replica` – Section of replica configurations. There can be multiple sections. - `replica` – Section of replica configurations. There can be multiple sections.
- `replica/host` – The MySQL host. - `replica/host` – The MySQL host.
\* `replica/priority` – The replica priority. When attempting to connect, ClickHouse traverses the replicas in order of priority. The lower the number, the higher the priority. \* `replica/priority` – The replica priority. When attempting to connect, ClickHouse traverses the replicas in order of priority. The lower the number, the higher the priority.
- `db` – Name of the database. - `db` – Name of the database.
- `table` – Name of the table. - `table` – Name of the table.
- `where ` – The selection criteria. Optional parameter. - `where ` – The selection criteria. Optional parameter.
- `invalidate_query` – Query for checking the dictionary status. Optional parameter. Read more in the section [Updating dictionaries](external_dicts_dict_lifetime.md#dicts-external_dicts_dict_lifetime). - `invalidate_query` – Query for checking the dictionary status. Optional parameter. Read more in the section [Updating dictionaries](external_dicts_dict_lifetime.md#dicts-external_dicts_dict_lifetime).
MySQL can be connected on a local host via sockets. To do this, set `host` and `socket`. MySQL can be connected on a local host via sockets. To do this, set `host` and `socket`.
Example of settings: Example of settings:
```xml ```xml
<source> <source>
<mysql> <mysql>
<host>localhost</host> <host>localhost</host>
<socket>/path/to/socket/file.sock</socket> <socket>/path/to/socket/file.sock</socket>
<user>clickhouse</user> <user>clickhouse</user>
<password>qwerty</password> <password>qwerty</password>
<db>db_name</db> <db>db_name</db>
<table>table_name</table> <table>table_name</table>
<where>id=10</where> <where>id=10</where>
<invalidate_query>SQL_QUERY</invalidate_query> <invalidate_query>SQL_QUERY</invalidate_query>
</mysql> </mysql>
</source> </source>
``` ```
<a name="dicts-external_dicts_dict_sources-clickhouse"></a> <a name="dicts-external_dicts_dict_sources-clickhouse"></a>
### ClickHouse ### ClickHouse
Example of settings: Example of settings:
```xml ```xml
<source> <source>
<clickhouse> <clickhouse>
<host>example01-01-1</host> <host>example01-01-1</host>
<port>9000</port> <port>9000</port>
<user>default</user> <user>default</user>
<password></password> <password></password>
<db>default</db> <db>default</db>
<table>ids</table> <table>ids</table>
<where>id=10</where> <where>id=10</where>
</clickhouse> </clickhouse>
</source> </source>
``` ```
Setting fields: Setting fields:
- `host` – The ClickHouse host. If it is a local host, the query is processed without any network activity. To improve fault tolerance, you can create a [Distributed](../../operations/table_engines/distributed.md#table_engines-distributed) table and enter it in subsequent configurations. - `host` – The ClickHouse host. If it is a local host, the query is processed without any network activity. To improve fault tolerance, you can create a [Distributed](../../operations/table_engines/distributed.md#table_engines-distributed) table and enter it in subsequent configurations.
- `port` – The port on the ClickHouse server. - `port` – The port on the ClickHouse server.
- `user` – Name of the ClickHouse user. - `user` – Name of the ClickHouse user.
- `password` – Password of the ClickHouse user. - `password` – Password of the ClickHouse user.
- `db` – Name of the database. - `db` – Name of the database.
- `table` – Name of the table. - `table` – Name of the table.
- `where ` – The selection criteria. May be omitted. - `where ` – The selection criteria. May be omitted.
- `invalidate_query` – Query for checking the dictionary status. Optional parameter. Read more in the section [Updating dictionaries](external_dicts_dict_lifetime.md#dicts-external_dicts_dict_lifetime). - `invalidate_query` – Query for checking the dictionary status. Optional parameter. Read more in the section [Updating dictionaries](external_dicts_dict_lifetime.md#dicts-external_dicts_dict_lifetime).
<a name="dicts-external_dicts_dict_sources-mongodb"></a> <a name="dicts-external_dicts_dict_sources-mongodb"></a>
### MongoDB ### MongoDB
Example of settings: Example of settings:
```xml ```xml
<source> <source>
<mongodb> <mongodb>
<host>localhost</host> <host>localhost</host>
<port>27017</port> <port>27017</port>
<user></user> <user></user>
<password></password> <password></password>
<db>test</db> <db>test</db>
<collection>dictionary_source</collection> <collection>dictionary_source</collection>
</mongodb> </mongodb>
</source> </source>
``` ```
Setting fields: Setting fields:
- `host` – The MongoDB host. - `host` – The MongoDB host.
- `port` – The port on the MongoDB server. - `port` – The port on the MongoDB server.
- `user` – Name of the MongoDB user. - `user` – Name of the MongoDB user.
- `password` – Password of the MongoDB user. - `password` – Password of the MongoDB user.
- `db` – Name of the database. - `db` – Name of the database.
- `collection` – Name of the collection. - `collection` – Name of the collection.
[Original article](https://clickhouse.yandex/docs/en/query_language/dicts/external_dicts_dict_sources/) <!--hide--> [Original article](https://clickhouse.yandex/docs/en/query_language/dicts/external_dicts_dict_sources/) <!--hide-->