thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-26 09:32:01 +00:00
Code Issues Packages Projects Releases Wiki Activity
158,687 Commits 9,177 Branches 1,781 Tags 2.1 GiB
03c5e4b107
Commit Graph

220 Commits

Author SHA1 Message Date
rfraposa
1b6916ddd2 Condensed dictionary docs into a single page 2023-02-28 12:01:52 -07:00
rfraposa
a4a5a8a7d3 Initial copy of doc-preview 2023-02-28 11:59:05 -07:00
rfraposa
e52edd4e85 Update external-dicts-dict-layout.md 2023-02-01 09:06:21 -07:00
Denny Crane
fda47bf4f8
Update external-dicts-dict-layout.md 2023-01-24 21:31:43 -04:00
Azat Khuzhin
4366f7fb3b Remove PREALLOCATE for HASHED/SPARSE_HASHED dictionaries 
It does not give significant benefit, but now, you hashed/sparse_hashed
dictionaries can be filled in parallel (#40003), using sharded
dictionaries, and this should be used instead of PREALLOCATE.

Note, that dictionaries, that had been created with PREALLOCATE will
work, but simply ignore this attribute.

Fixes: #41985 (cc @alexey-milovidov)
Reverts: #23979 (cc @kitaisreal)

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-01-18 20:18:37 +01:00
Azat Khuzhin
99063b152f Allow to configure queue backlog of the parallel hashed dictionary loader 
v2: Decrease default parallel_queue_backlog to 10000 (same speed)
v3: Rename parallel_queue_backlog to per_shard_load_backlog
v3: Rename per_shard_load_backlog to shard_load_queue_backlog
v4: Fix documentation
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-01-13 13:39:26 +01:00
Azat Khuzhin
345c422e28 Add ability to load hashed dictionaries using multiple threads 
Right now dictionaries (here I will talk about only
HASHED/SPARSE_HASHED/COMPLEX_KEY_HASHED/COMPLEX_KEY_SPARSE_HASHED)
can load data only in one thread, since it uses one hash table that
cannot be filled from multiple threads.

And in case you have very big dictionary (i.e. 10e9 elements), it can
take a awhile to load them, especially for SPARSE_HASHED variants (and
if you have such amount of elements there, you are likely use
SPARSE_HASHED, since it requires less memory), in my env it takes ~4
hours, which is enormous amount of time.

So this patch add support of shards for dictionaries, number of shards
determine how much hash tables will use this dictionary, also, and which
is more important, how much threads it can use to load the data.

And with 16 threads this works 2x faster, not perfect though, see the
follow up patches in this series.

v0: PARTITION BY
v1: SHARDS 1
v2: SHARDS(1)
v3: tried optimized mod - logical and, but it does not gain even 10%
v4: tried squashing more (max_block_size * shards), but it does not gain even 10% either
v5: move SHARDS into layout parameters (unknown simply ignored)
v6: tune params for perf tests (to avoid too long queries)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-01-13 13:39:25 +01:00
Han Fei
6ed4570f73
Merge branch 'master' into regexp-tree-dictionary 2023-01-10 15:36:30 +01:00
Han Fei
5f8296b719
Update docs/en/sql-reference/dictionaries/external-dictionaries/regexp-tree.md 
Co-authored-by: Vladimir C <vdimir@clickhouse.com>
 2023-01-10 14:41:06 +01:00
Ivan Blinkov
61c2f23713 Remove leftover empty lines at the end of markdown files 2023-01-09 15:15:18 +01:00
Han Fei
f2a9eea995 write docs and optimize regex compile 2023-01-05 17:38:01 +01:00
Denny Crane
850f77f4d2
Update external-dicts-dict-sources.md 2022-12-26 16:21:36 -04:00
Dale Mcdiarmid
1f5e6799ec revert contents change 2022-12-16 12:03:57 +00:00
Dale Mcdiarmid
ba52210124 revert format issue 2022-12-16 12:00:12 +00:00
Dale Mcdiarmid
22e8477b2a cross link dictionaries + udf posts£ 2022-12-14 15:01:15 +00:00
Dan Roscigno
c5eb269515
Merge pull request #43943 from DanRoscigno/update-operations-docs 
Update operations docs
 2022-12-05 20:58:42 -05:00
Dale Mcdiarmid
5ab5aa13f4 cross link docs to blogs 2022-12-05 17:28:03 +00:00
DanRoscigno
08e8ea1bfa update link 2022-12-05 08:23:51 -05:00
DanRoscigno
5e087ae967 link to tutorial 2022-11-16 11:54:06 -05:00
DanRoscigno
c60b98f576 updates from review 2022-11-15 16:17:43 -05:00
DanRoscigno
63ae261119 move tip to snippet 2022-11-15 12:44:54 -05:00
DanRoscigno
5aae1d0724 update note ot include default user 2022-11-15 10:49:13 -05:00
DanRoscigno
34f90ff6ef update pages that refer to dictionaries 2022-11-07 09:26:50 -05:00
DanRoscigno
7271d4d122 add note to embedded dict 2022-11-02 17:52:35 -04:00
DanRoscigno
df4868587b add note to use DDL 2022-11-02 17:40:55 -04:00
DanRoscigno
3dd5df4615 update examples 2022-11-02 16:58:07 -04:00
DanRoscigno
c8f9af1afa start renaming 2022-11-02 15:47:11 -04:00
Camilo Sierra
b6d84ef5fa
range hashed missing parameters 
On the query we have missing two options:
- `LIFETIME` Is not on the example if you copy paste you will have an Exception `DB::Exception: Cannot create dictionary with empty lifetime.`
- `SOURCE` was not mentioned and it's important to link to the main/source table.
- There was an error on the `dictGetT` function there was an additional T this function do not exist (I have tested and we need to use `dictGet`).
- Also in the Dictionary example we have no extra attribute other than the id and the two dates, and for running the queries and the `dicGet` function you need an additional attribute this is why I have added `advertiser_id` (BTW I use advertiser_id as this was use in the example just before) and also add one example, without the example it was not easy to understand what was the 'attr_name' mentioned before.
- I add an example as an user did not knew how to cast the date to a Uint64 (Because most of the time the original/raw dates are defined on the range as Date64, so this example will explain them how to cast when doing the query)
 2022-11-02 15:43:57 +01:00
UnamedRus
d4c6268806
Fix Polygon dict xml config 
Missing  <attribute> tag
 2022-10-28 18:47:32 +04:00
Alexey Milovidov
deeaea004d
Merge pull request #40295 from cyber-moon/patch-1 
Add description of {condition}-keyword
 2022-09-04 04:57:30 +03:00
Alexey Milovidov
0190c56faf
Merge pull request #40770 from den-crane/patch-40 
Doc. Fix cache dictionaries doc.
 2022-08-30 02:19:32 +03:00
Denny Crane
fe0f18f21d
Update external-dicts-dict-layout.md 2022-08-29 15:19:15 -03:00
DanRoscigno
c4b8137d31 replace symlinks 2022-08-29 12:19:50 -04:00
DanRoscigno
5b5fcc56aa add slugs 2022-08-28 10:53:34 -04:00
Stephan
9242c2adb4
Add description of {condition}-keyword 
Using the {condition}-keyword for Dictionaries is supported since a few weeks (see https://github.com/ClickHouse/ClickHouse/issues/33746 ). This PR adds a corresponding documentation.
 2022-08-17 11:56:48 +02:00
DanRoscigno
c411763c2f split the note into two notes 2022-07-20 09:41:00 -04:00
Nikita Taranov
7f6b175c3e impl 2022-07-18 19:39:10 +02:00
rfraposa
869967de41 Remove H1 anchor tags from docs 2022-06-02 04:55:18 -06:00
Denny Crane
d7e098fb17
Update external-dicts-dict-layout.md 2022-05-28 17:26:11 -03:00
Maksim Kita
fc4f241f01 Fix documentation for executable user defined functions configuration 2022-04-28 18:01:10 +02:00
rfraposa
6576fd88c1 EOL fixes 2022-04-19 13:50:45 -06:00
rfraposa
d9e6f780db Fixed broken links 2022-04-10 23:01:34 -06:00
rfraposa
8f01fe9c49 Revised /en folder 2022-04-09 07:34:21 -06:00
rfraposa
5250d9ad11 Removed /ja folder, cleaned up /ru markdown 2022-04-09 07:29:05 -06:00
Alexey Milovidov
9854b55835
Revert "Format changes for new docs" 2022-04-04 02:05:35 +03:00
rfraposa
72a00e2c62 Fixed broken links 2022-04-02 17:38:54 -06:00
rfraposa
c0a791a784 Fixing conflicts with source repo 2022-04-01 11:15:41 -06:00
rfraposa
560471f991 Update /sql-reference docs 2022-03-29 22:06:21 -06:00
rfraposa
33e28bcb18 Updated /development folder 2022-03-29 08:26:35 -06:00
Xin Wang
56aa3661ae
fix: modify dictGet* functions 
1. changes the `dictGetT` to `dictGet*` to avoid confusion
2. modify the format of tables for alignment.
 2022-03-18 17:02:13 +08:00
Rich Raposa
6fbb63b30c
Update internal-dicts.md 
The mentioned functions have already been removed
 2022-03-13 21:31:48 -06:00
rfraposa
5a4466cec7 Update references in docs 2022-03-12 00:24:31 -06:00
Nikita
d6147f7105
Removed line break for correct table display 
https://clickhouse.com/docs/en/sql-reference/dictionaries/external-dictionaries/external-dicts-dict-structure/#ext_dict_structure-attributes
 2022-01-25 01:29:39 +03:00
Maksim Kita
21ace30a01 Updated documentation 2022-01-22 18:27:55 +00:00
Maksim Kita
af37eca3fb Executable UserDefinedFunction documentation fix 2022-01-07 19:19:18 +03:00
Maksim Kita
8e83f2d154 Updated documentation 2022-01-02 20:02:53 +03:00
Maksim Kita
49d4619890 Updated docs 2021-12-28 22:55:31 +03:00
Xin Wang
3edb2ca806
Update external-dicts-dict-polygon.md 
fix a typo `tne` -> `the`
 2021-12-20 16:32:38 +08:00
Maksim Kita
e25ac911af
Merge pull request #31458 from ka1bi4/romanzhukov-DOCSUP-16661-polygon-dictionary 
DOCSUP-16661: [CLICKHOUSEDOCS] Polygon dictionary support read
 2021-12-01 01:12:53 +03:00
romanzhukov
d9b848f99d DOCSUP-16661: Update example and docs. 2021-11-20 14:28:25 +03:00
romanzhukov
3f1150df2d DOCSUP-16661: Update example and docs. 2021-11-20 14:15:39 +03:00
Andrey Torsunov
7f25ac0598
typo in PostgreSQL 2021-11-19 14:37:10 +03:00
romanzhukov
5769631ec1 DOCSUP-16661: Update docs. 2021-11-16 23:45:53 +03:00
sevirov
2cf825d1d1
Update docs/en/sql-reference/dictionaries/external-dictionaries/external-dicts-dict-sources.md 
Co-authored-by: gyuton <40863448+gyuton@users.noreply.github.com>
 2021-11-09 19:11:51 +03:00
sevirov
86b2a5f411
Update docs/en/sql-reference/dictionaries/external-dictionaries/external-dicts-dict-sources.md 
Co-authored-by: gyuton <40863448+gyuton@users.noreply.github.com>
 2021-11-09 19:11:45 +03:00
sevirov
8c5378a0ea
Update docs/en/sql-reference/dictionaries/external-dictionaries/external-dicts-dict-sources.md 
Co-authored-by: gyuton <40863448+gyuton@users.noreply.github.com>
 2021-11-09 19:11:39 +03:00
sevirov
30ae146573
Update docs/en/sql-reference/dictionaries/external-dictionaries/external-dicts-dict-sources.md 
Co-authored-by: gyuton <40863448+gyuton@users.noreply.github.com>
 2021-11-09 19:11:33 +03:00
sevirov
40b693e14b
Update docs/en/sql-reference/dictionaries/external-dictionaries/external-dicts-dict-sources.md 
Co-authored-by: gyuton <40863448+gyuton@users.noreply.github.com>
 2021-11-09 19:11:25 +03:00
Dmitriy
ab3dbd8108 Translate to Russian 
Выполнил перевод на русский язык.
 2021-11-06 14:24:01 +03:00
Dmitriy
e55bfe7f37 Update external-dicts-dict-sources.md 
Поправил описание.
 2021-11-03 17:40:49 +03:00
Dmitriy
d4eb5ec1d0 Merge remote-tracking branch 'remotes/clickhouse/master' into sevirov-DOCSUP-14511-document_custom_query_for_database_dictionaries 2021-10-31 21:32:42 +03:00
Dmitriy
43d68af165 Add the custom query 
Добавил поле query в некоторые источники словарей.
 2021-10-31 20:53:49 +03:00
Azat Khuzhin
e0cd7a8304 docs: switch <yandex> to <clickhouse> 2021-10-26 08:52:29 +03:00
Maksim Kita
613a2fc46c
Merge pull request #30409 from sevirov/sevirov-DOCSUP-16821-edit_and_translate_to_russian_hashedarray_dictionary 
DOCSUP-16821: Edit and translate to Russian (HashedArray dictionary)
 2021-10-23 00:15:35 +03:00
Dmitriy
e3927fa9c8 Fix links 
Поправил ссылки.
 2021-10-22 19:36:29 +03:00
alexey-milovidov
7023423f66
Merge pull request #28705 from gyuton/gyuton-DOCSUP-14093-Edit-and-translate-to-Russian 
DOCSUP-14093: Edited and translated to Russian (external dicts)
 2021-10-21 23:33:41 +03:00
Dmitriy
377f6cf3c2 Edit and translate to Russian 
Выполнил перевод на русский язык.
 2021-10-19 21:51:46 +03:00
Denny Crane
eb66442feb
Update external-dicts-dict-layout.md 2021-10-17 16:21:21 -03:00
Maksim Kita
1b50d26c51 Added documentation 2021-10-16 13:41:56 +03:00
Maksim Kita
7e23a7d814 RangeHashedDictionary range type docs fix 2021-09-15 12:23:31 +03:00
George
3ce7331211 Edited and translated to Russian 2021-09-07 20:07:27 +03:00
Dmitriy
7021d9bbe4
Update docs/en/sql-reference/dictionaries/external-dictionaries/external-dicts-dict-layout.md 
Co-authored-by: Anna <42538400+adevyatova@users.noreply.github.com>
 2021-08-26 19:47:21 +03:00
Dmitriy
c75d776d04 Document the ComplexKeyRangeHashed dictionary 
Задокументировал новый словарь.
 2021-08-21 22:44:12 +03:00
Maksim Kita
3b4eb2d769 Fixed code review issues 2021-08-03 00:53:43 +03:00
Maksim Kita
d2440dca86 Updated cache dictionaries documentation 2021-08-02 15:18:05 +03:00
Alexey Milovidov
d87d833691 Remove trailing whitespaces from docs 2021-07-29 18:20:55 +03:00
romanzhukov
69f3804980 DOCSUP-11554: dictionary source secure setting added 2021-07-18 19:06:22 +03:00
Maksim Kita
39c5ba40f7
Merge pull request #26055 from kitaisreal/clickhouse-dictionary-source-secure-setting-added-documentation 
ClickHouse dictionary source secure setting added documentation
 2021-07-07 23:49:42 +03:00
Maksim Kita
9d0c2f83f8 ClickHouse dictionary source secure setting added documentation 2021-07-07 22:14:30 +03:00
Evgeniia Sudarikova
0955d001cd added array to types 2021-07-04 16:11:29 +03:00
tavplubix
56da480497
Merge pull request #25302 from otrazhenia/evgsudarikova-DOCSUP-10299 
DOCSUP-10299: Edit and translate to Russian
 2021-06-30 14:02:43 +03:00
Evgenia Sudarikova
2bb9bb6a86
Update docs/en/sql-reference/dictionaries/external-dictionaries/external-dicts-dict-lifetime.md 
Co-authored-by: olgarev <56617294+olgarev@users.noreply.github.com>
 2021-06-24 12:53:39 +03:00
Evgenia Sudarikova
c7d5282798
Update docs/en/sql-reference/dictionaries/external-dictionaries/external-dicts-dict-lifetime.md 
Co-authored-by: olgarev <56617294+olgarev@users.noreply.github.com>
 2021-06-24 12:53:25 +03:00
Kseniia Sumarokova
84f317cc79
Merge pull request #25439 from sevirov/sevirov-DOCSUP-8661-mysql_dictionary_source 
DOCSUP-8661: MySQL dictionary source: the fail_on_connection_loss config parameter.
 2021-06-23 12:37:21 +03:00
Dmitriy
8091bd382d The small fix 
Внес небольшую поправку.
 2021-06-22 19:15:19 +03:00
Dmitriy
ee28db0daf Fix the inaccuracy 
Внес небольшую поправку.
 2021-06-22 00:23:24 +03:00
sevirov
cbca2a6c36
Update docs/en/sql-reference/dictionaries/external-dictionaries/external-dicts-dict-sources.md 
Co-authored-by: olgarev <56617294+olgarev@users.noreply.github.com>
 2021-06-21 20:11:47 +03:00
Dmitriy
ccad32fe96 Translate to Russian 
Выполнил перевод на русский язык.
 2021-06-18 21:20:53 +03:00
Dmitriy
d69790945d Add the fail_on_connection_loss configuration parameter 
Добавил описание параметра fail_on_connection_loss.
 2021-06-17 23:29:01 +03:00
George
864b35533e small fixes 2021-06-16 13:53:09 +03:00