thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-09-19 16:20:50 +00:00
Code Issues Packages Projects Releases Wiki Activity

Optimize the merge if all hashSets are singleLevel in UniqExactSet (#52973)

Browse Source 
* Optimize the merge if all hashSets are singleLevel

In PR(https://github.com/ClickHouse/ClickHouse/pull/50748), it has added new phase
`parallelizeMergePrepare` before merge if all the hashSets are not all singleLevel
or not all twoLevel. Then it will convert all the singleLevelSet to twoLevelSet in
parallel, which will increase the CPU utilization and QPS.

But if all the hashtables are singleLevel, it could also benefit from the
`parallelizeMergePrepare` optimization in most cases if the hashtable size are not
too small. By tuning the Query `SELECT COUNT(DISTINCT SearchPhase) FROM hits_v1`
in different threads, we have got the mild threshold 6,000.

Test patch with the Query 'SELECT COUNT(DISTINCT Title) FROM hits_v1' on 2x80 vCPUs
server. If the threads are less than 48, the hashSets are all twoLevel or mixed by
singleLevel and twoLevel. If the threads are over 56, all the hashSets are singleLevel.
And the QPS has got at most 2.35x performance gain.

Threads	Opt/Base
8	100.0%
16	99.4%
24	110.3%
32	99.9%
40	99.3%
48	99.8%
56	183.0%
64	234.7%
72	233.1%
80	229.9%
88	224.5%
96	229.6%
104	235.1%
112	229.5%
120	229.1%
128	217.8%
136	222.9%
144	217.8%
152	204.3%
160	203.2%

Signed-off-by: Jiebin Sun <jiebin.sun@intel.com>

* Add the comment and explanation for PR#52973

Signed-off-by: Jiebin Sun <jiebin.sun@intel.com>

---------

Signed-off-by: Jiebin Sun <jiebin.sun@intel.com>
This commit is contained in:
Jiebin Sun 2023-08-30 17:26:16 +08:00 committed by GitHub
parent 36fb7cfbd1
commit 7c529e5691
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 14 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

13
src/AggregateFunctions/UniqExactSet.h
View File

@ -34,6 +34,7 @@ public:
static void parallelizeMergePrepare(const std::vector<UniqExactSet *> & data_vec, ThreadPool & thread_pool)
{
unsigned long single_level_set_num = 0;
unsigned long all_single_hash_size = 0;
for (auto ele : data_vec)
{
@ -41,7 +42,17 @@ public:
single_level_set_num ++;
}
if (single_level_set_num > 0 && single_level_set_num < data_vec.size())
if (single_level_set_num == data_vec.size())
{
for (auto ele : data_vec)
all_single_hash_size += ele->size();
}
/// If all the hashtables are mixed by singleLevel and twoLevel, or all singleLevel (larger than 6000 for average value), they could be converted into
/// twoLevel hashtables in parallel and then merge together. please refer to the following PR for more details.
/// https://github.com/ClickHouse/ClickHouse/pull/50748
/// https://github.com/ClickHouse/ClickHouse/pull/52973
if ((single_level_set_num > 0 && single_level_set_num < data_vec.size()) || ((all_single_hash_size/data_vec.size()) > 6000))
{
try
{

2
tests/performance/prepare_hash_before_merge.xml
View File

@ -1,4 +1,6 @@
<test>
<query>SELECT COUNT(DISTINCT Title) FROM test.hits SETTINGS max_threads = 24</query>
<query>SELECT COUNT(DISTINCT Title) FROM test.hits SETTINGS max_threads = 56</query>
<query>SELECT COUNT(DISTINCT Title) FROM test.hits SETTINGS max_threads = 64</query>
<query>SELECT COUNT(DISTINCT Referer) FROM test.hits SETTINGS max_threads = 22</query>
</test>