# Sources of External Dictionaries {#dicts-external_dicts_dict_sources} An external dictionary can be connected from many different sources. If dictionary is configured using xml-file, the configuration looks like this: ```xml ... ... ... ``` In case of [DDL-query](../create.md#create-dictionary-query), equal configuration will looks like: ```sql CREATE DICTIONARY dict_name (...) ... SOURCE(SOURCE_TYPE(param1 val1 ... paramN valN)) -- Source configuration ... ``` The source is configured in the `source` section. Types of sources (`source_type`): - [Local file](#dicts-external_dicts_dict_sources-local_file) - [Executable file](#dicts-external_dicts_dict_sources-executable) - [HTTP(s)](#dicts-external_dicts_dict_sources-http) - DBMS - [ODBC](#dicts-external_dicts_dict_sources-odbc) - [MySQL](#dicts-external_dicts_dict_sources-mysql) - [ClickHouse](#dicts-external_dicts_dict_sources-clickhouse) - [MongoDB](#dicts-external_dicts_dict_sources-mongodb) - [Redis](#dicts-external_dicts_dict_sources-redis) ## Local File {#dicts-external_dicts_dict_sources-local_file} Example of settings: ```xml /opt/dictionaries/os.tsv TabSeparated ``` or ```sql SOURCE(FILE(path '/opt/dictionaries/os.tsv' format 'TabSeparated')) ``` Setting fields: - `path` – The absolute path to the file. - `format` – The file format. All the formats described in "[Formats](../../interfaces/formats.md#formats)" are supported. ## Executable File {#dicts-external_dicts_dict_sources-executable} Working with executable files depends on [how the dictionary is stored in memory](external_dicts_dict_layout.md). If the dictionary is stored using `cache` and `complex_key_cache`, ClickHouse requests the necessary keys by sending a request to the executable file's STDIN. Otherwise, ClickHouse starts executable file and treats its output as dictionary data. Example of settings: ```xml cat /opt/dictionaries/os.tsv TabSeparated ``` or ```sql SOURCE(EXECUTABLE(command 'cat /opt/dictionaries/os.tsv' format 'TabSeparated')) ``` Setting fields: - `command` – The absolute path to the executable file, or the file name (if the program directory is written to `PATH`). - `format` – The file format. All the formats described in "[Formats](../../interfaces/formats.md#formats)" are supported. ## HTTP(s) {#dicts-external_dicts_dict_sources-http} Working with an HTTP(s) server depends on [how the dictionary is stored in memory](external_dicts_dict_layout.md). If the dictionary is stored using `cache` and `complex_key_cache`, ClickHouse requests the necessary keys by sending a request via the `POST` method. Example of settings: ```xml http://[::1]/os.tsv TabSeparated user password
API-KEY key
``` or ```sql SOURCE(HTTP( url 'http://[::1]/os.tsv' format 'TabSeparated' credentials(user 'user' password 'password') headers(header(name 'API-KEY' value 'key')) )) ``` In order for ClickHouse to access an HTTPS resource, you must [configure openSSL](../../operations/server_settings/settings.md#server_settings-openssl) in the server configuration. Setting fields: - `url` – The source URL. - `format` – The file format. All the formats described in "[Formats](../../interfaces/formats.md#formats)" are supported. - `credentials` – Basic HTTP authentication. Optional parameter. - `user` – Username required for the authentication. - `password` – Password required for the authentication. - `headers` – All custom HTTP headers entries used for the HTTP request. Optional parameter. - `header` – Single HTTP header entry. - `name` – Identifiant name used for the header send on the request. - `value` – Value set for a specific identifiant name. ## ODBC {#dicts-external_dicts_dict_sources-odbc} You can use this method to connect any database that has an ODBC driver. Example of settings: ```xml DatabaseName ShemaName.TableName
DSN=some_parameters SQL_QUERY
``` or ```sql SOURCE(ODBC( db 'DatabaseName' table 'SchemaName.TableName' connection_string 'DSN=some_parameters' invalidate_query 'SQL_QUERY' )) ``` Setting fields: - `db` – Name of the database. Omit it if the database name is set in the `` parameters. - `table` – Name of the table and schema if exists. - `connection_string` – Connection string. - `invalidate_query` – Query for checking the dictionary status. Optional parameter. Read more in the section [Updating dictionaries](external_dicts_dict_lifetime.md). ClickHouse receives quoting symbols from ODBC-driver and quote all settings in queries to driver, so it's necessary to set table name accordingly to table name case in database. If you have a problems with encodings when using Oracle, see the corresponding [FAQ](../../faq/general.md#oracle-odbc-encodings) article. ### Known vulnerability of the ODBC dictionary functionality !!! attention When connecting to the database through the ODBC driver connection parameter `Servername` can be substituted. In this case values of `USERNAME` and `PASSWORD` from `odbc.ini` are sent to the remote server and can be compromised. **Example of insecure use** Let's configure unixODBC for PostgreSQL. Content of `/etc/odbc.ini`: ```text [gregtest] Driver = /usr/lib/psqlodbca.so Servername = localhost PORT = 5432 DATABASE = test_db #OPTION = 3 USERNAME = test PASSWORD = test ``` If you then make a query such as ```sql SELECT * FROM odbc('DSN=gregtest;Servername=some-server.com', 'test_db'); ``` ODBC driver will send values of `USERNAME` and `PASSWORD` from `odbc.ini` to `some-server.com`. ### Example of Connecting PostgreSQL Ubuntu OS. Installing unixODBC and the ODBC driver for PostgreSQL: ```bash $ sudo apt-get install -y unixodbc odbcinst odbc-postgresql ``` Configuring `/etc/odbc.ini` (or `~/.odbc.ini`): ```text [DEFAULT] Driver = myconnection [myconnection] Description = PostgreSQL connection to my_db Driver = PostgreSQL Unicode Database = my_db Servername = 127.0.0.1 UserName = username Password = password Port = 5432 Protocol = 9.3 ReadOnly = No RowVersioning = No ShowSystemTables = No ConnSettings = ``` The dictionary configuration in ClickHouse: ```xml table_name DSN=myconnection postgresql_table
300 360 id some_column UInt64 0
``` or ```sql CREATE DICTIONARY table_name ( id UInt64, some_column UInt64 DEFAULT 0 ) PRIMARY KEY id SOURCE(ODBC(connection_string 'DSN=myconnection' table 'postgresql_table')) LAYOUT(HASHED()) LIFETIME(MIN 300 MAX 360) ``` You may need to edit `odbc.ini` to specify the full path to the library with the driver `DRIVER=/usr/local/lib/psqlodbcw.so`. ### Example of Connecting MS SQL Server Ubuntu OS. Installing the driver: : ```bash $ sudo apt-get install tdsodbc freetds-bin sqsh ``` Configuring the driver: ```bash $ cat /etc/freetds/freetds.conf ... [MSSQL] host = 192.168.56.101 port = 1433 tds version = 7.0 client charset = UTF-8 $ cat /etc/odbcinst.ini ... [FreeTDS] Description = FreeTDS Driver = /usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so Setup = /usr/lib/x86_64-linux-gnu/odbc/libtdsS.so FileUsage = 1 UsageCount = 5 $ cat ~/.odbc.ini ... [MSSQL] Description = FreeTDS Driver = FreeTDS Servername = MSSQL Database = test UID = test PWD = test Port = 1433 ``` Configuring the dictionary in ClickHouse: ```xml test dict
DSN=MSSQL;UID=test;PWD=test
300 360 k s String
``` or ```sql CREATE DICTIONARY test ( k UInt64, s String DEFAULT '' ) PRIMARY KEY k SOURCE(ODBC(table 'dict' connection_string 'DSN=MSSQL;UID=test;PWD=test')) LAYOUT(FLAT()) LIFETIME(MIN 300 MAX 360) ``` ## DBMS ### MySQL {#dicts-external_dicts_dict_sources-mysql} Example of settings: ```xml 3306 clickhouse qwerty example01-1 1 example01-2 1 db_name table_name
id=10 SQL_QUERY
``` or ```sql SOURCE(MYSQL( port 3306 user 'clickhouse' password 'qwerty' replica(host 'example01-1' priority 1) replica(host 'example01-2' priority 1) db 'db_name' table 'table_name' where 'id=10' invalidate_query 'SQL_QUERY' )) ``` Setting fields: - `port` – The port on the MySQL server. You can specify it for all replicas, or for each one individually (inside ``). - `user` – Name of the MySQL user. You can specify it for all replicas, or for each one individually (inside ``). - `password` – Password of the MySQL user. You can specify it for all replicas, or for each one individually (inside ``). - `replica` – Section of replica configurations. There can be multiple sections. - `replica/host` – The MySQL host. - `replica/priority` – The replica priority. When attempting to connect, ClickHouse traverses the replicas in order of priority. The lower the number, the higher the priority. - `db` – Name of the database. - `table` – Name of the table. - `where ` – The selection criteria. The syntax for conditions is the same as for `WHERE` clause in MySQL, for example, `id > 10 AND id < 20`. Optional parameter. - `invalidate_query` – Query for checking the dictionary status. Optional parameter. Read more in the section [Updating dictionaries](external_dicts_dict_lifetime.md). MySQL can be connected on a local host via sockets. To do this, set `host` and `socket`. Example of settings: ```xml localhost /path/to/socket/file.sock clickhouse qwerty db_name table_name
id=10 SQL_QUERY
``` or ```sql SOURCE(MYSQL( host 'localhost' socket '/path/to/socket/file.sock' user 'clickhouse' password 'qwerty' db 'db_name' table 'table_name' where 'id=10' invalidate_query 'SQL_QUERY' )) ``` ### ClickHouse {#dicts-external_dicts_dict_sources-clickhouse} Example of settings: ```xml example01-01-1 9000 default default ids
id=10
``` or ```sql SOURCE(CLICKHOUSE( host 'example01-01-1' port 9000 user 'default' password '' db 'default' table 'ids' where 'id=10' )) ``` Setting fields: - `host` – The ClickHouse host. If it is a local host, the query is processed without any network activity. To improve fault tolerance, you can create a [Distributed](../../operations/table_engines/distributed.md) table and enter it in subsequent configurations. - `port` – The port on the ClickHouse server. - `user` – Name of the ClickHouse user. - `password` – Password of the ClickHouse user. - `db` – Name of the database. - `table` – Name of the table. - `where ` – The selection criteria. May be omitted. - `invalidate_query` – Query for checking the dictionary status. Optional parameter. Read more in the section [Updating dictionaries](external_dicts_dict_lifetime.md). ### MongoDB {#dicts-external_dicts_dict_sources-mongodb} Example of settings: ```xml localhost 27017 test dictionary_source ``` or ```sql SOURCE(MONGO( host 'localhost' port 27017 user '' password '' db 'test' collection 'dictionary_source' )) ``` Setting fields: - `host` – The MongoDB host. - `port` – The port on the MongoDB server. - `user` – Name of the MongoDB user. - `password` – Password of the MongoDB user. - `db` – Name of the database. - `collection` – Name of the collection. ### Redis {#dicts-external_dicts_dict_sources-redis} Example of settings: ```xml localhost 6379 simple 0 ``` or ```sql SOURCE(REDIS( host 'localhost' port 6379 storage_type 'simple' db_index 0 )) ``` Setting fields: - `host` – The Redis host. - `port` – The port on the Redis server. - `storage_type` – The structure of internal Redis storage using for work with keys. `simple` is for simple sources and for hashed single key sources, `hash_map` is for hashed sources with two keys. Ranged sources and cache sources with complex key are unsupported. May be omitted, default value is `simple`. - `db_index` – The specific numeric index of Redis logical database. May be omitted, default value is 0. [Original article](https://clickhouse.tech/docs/en/query_language/dicts/external_dicts_dict_sources/)