thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 02:41:59 +00:00
Code Issues Packages Projects Releases Wiki Activity
160,198 Commits 9,328 Branches 1,783 Tags 2.1 GiB
68e77f6f33
Commit Graph

160198 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Azat Khuzhin
b0da170e62 Fix default value for group_by_overflow_mode 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-11-28 16:01:53 +01:00
Azat Khuzhin
a8eaf3ecde Fix copy-paste typo for max_bytes_ratio_before_external_group_by doc 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-11-28 16:01:53 +01:00
Azat Khuzhin
bdb39a6fef Change logic of automatic ORDER/GROUP BY to be based on available memory 
- remove max_bytes_ratio_before_external_{order,group_by}_for_server
- change the way max_bytes_ratio_before_external_{order,group_by} works

Note, that it is not enough to transform ratio to bytes in
executeQuery(), since in this case it will not work for merges and
internal queries, plus, you have to reset them for Distributed engine
and update it for Merge/View/...

This patch also introduce some helpers (see MemoryTrackerUtils) and
adjust Aggregator::Params constructor to accept Settings object instead
of tons of arguments.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-11-28 16:01:53 +01:00
Azat Khuzhin
14e27b7de9 Move settings changes to 24.12 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-11-28 16:01:53 +01:00
Azat Khuzhin
ee271c2a06 Reduce copy-paste by introducing Aggregator::writeToTemporaryFileIfNeeded() 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-11-28 16:01:53 +01:00
Azat Khuzhin
520b21efa5 Respect max_bytes_ratio_before_external_sort_for_server in Aggregator::mergeOnBlock() 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-11-28 16:01:53 +01:00
Azat Khuzhin
2d3d0bb5cc Respect max_bytes_ratio_before_external_sort_for_server for ORDER BY storage check 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-11-28 16:01:53 +01:00
Azat Khuzhin
2fab184ee4 Properly handle unlimited memory limit for automatic GROUP/ORDER BY 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-11-28 16:01:53 +01:00
Azat Khuzhin
db1f459cc4 tests: disable test_max_bytes_ratio_before_external_order_group_by_for_server under TSan 
The problem is that TSan has some memory overhead which increases
process RSS to 10Gi in the middle of the query and it will fail.
Another option is to avoid syncing with RSS too frequently, but I doubt
that it is significant to run this test under TSan

v1: increase memory to 20Gi
    Fixes: https://s3.amazonaws.com/clickhouse-test-reports/71406/b33be86dee3c7616e9193c339837b9e250810557/integration_tests__tsan__[6_6].html
v2: instead just disable under TSan
    Fixes: https://s3.amazonaws.com/clickhouse-test-reports/71406/df68cf7362d825c3d83991fd74c391a49e73a8b1/integration_tests__tsan__[6_6].html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-11-28 16:01:53 +01:00
Azat Khuzhin
02aa6eac82 Cover max_bytes_ratio_before_external_{order,group_by}_for_server 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-11-28 16:01:53 +01:00
Azat Khuzhin
909eef401a Implement max_bytes_ratio_before_external_sort_for_server 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-11-28 16:01:53 +01:00
Azat Khuzhin
d5a990346c Implement max_bytes_ratio_before_external_group_by_for_server 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-11-28 16:01:53 +01:00
Azat Khuzhin
378b6c80a6 Automatic ORDER BY to disk (max_bytes_ratio_before_external_sort) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-11-28 16:01:53 +01:00
Azat Khuzhin
a856c30c7c Automatic GROUP BY to disk (max_bytes_ratio_before_external_group_by) 
Simpler then max_bytes_before_external_group_by, that allows you not to
think about memory consumption of the query, and can be set globally to
i.e. 0.5 so that the external aggregation will be enabled once the
memory usage reaches the 50%.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-11-28 16:01:53 +01:00
Azat Khuzhin
c29907c009 Introduce getCurrentQueryMemoryUsage() helper for Aggregator 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-11-28 16:01:53 +01:00
Azat Khuzhin
1b27a17959 Add annotation for calling constructor of Aggregator::Params 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-11-28 16:01:53 +01:00
Vitaly Baranov
13d4c16a56
Merge pull request #72427 from vitlibar/parallel-creating-tables-while-restoring 
Implement creating tables in parallel while restoring from a backup
 2024-11-28 14:48:39 +00:00
Nikita Taranov
863f3fba79
Merge branch 'master' into raise_default_for_prealloc_in_ht_optimisation 2024-11-28 15:32:21 +01:00
Anton Popov
5b03379aba
Merge pull request #72479 from CurtizJ/randomize-primary-key-settings 
Randomize settings for primary key cache
 2024-11-28 14:30:03 +00:00
alesapin
79f2bfc901
Update SerializationAggregateFunction.cpp 2024-11-28 15:20:51 +01:00
Antonio Andelic
5bdd41cfc0 Fix 2024-11-28 15:09:49 +01:00
Alexey Milovidov
4bbadf64c6
Merge pull request #72109 from ClickHouse/pr-skip-index-analysis-on-workers 
PR: do index analysis only on coordinator
 2024-11-28 12:49:28 +00:00
Jiebin Sun
0bbe51d4e9
Merge branch 'ClickHouse:master' into shared_mutex_optimization 2024-11-28 20:48:21 +08:00
Alexey Milovidov
c8942bb213
Merge pull request #72558 from azat/ci/faster-shutdown 
Fix possible CI failures due to slow logs proxy tables shutdown
 2024-11-28 12:48:14 +00:00
Alexey Milovidov
17165c47c9
Merge pull request #72575 from nauu/update_trusted_contributors 
Update trusted contributors
 2024-11-28 12:32:40 +00:00
Vladimir Cherkasov
3dfa9648a7
Merge pull request #72574 from tbragin/patch-23 
Update README.md - Update Meetups
 2024-11-28 12:04:44 +00:00
vdimir
7ae8a1fa4a
up test 2024-11-28 11:51:47 +00:00
Sema Checherinda
15e9009ccb fix pipeline cacelation 2024-11-28 12:46:57 +01:00
vdimir
e875005567
Fix data race in Squashing with ColumnConst 2024-11-28 11:44:08 +00:00
Robert Schulze
e55705978d
Update LeastGreatestGeneric.h 2024-11-28 12:33:06 +01:00
Robert Schulze
021379bc35
Update LeastGreatestGeneric.h 2024-11-28 12:31:03 +01:00
Robert Schulze
cc14a2646c
Update LeastGreatestGeneric.h 2024-11-28 12:22:06 +01:00
Robert Schulze
db065b43c1
Update LeastGreatestGeneric.h 2024-11-28 12:21:27 +01:00
Robert Schulze
af0709aa7f
Update 03174_least_greatest_ignore_null_input_values.reference 2024-11-28 11:59:18 +01:00
Robert Schulze
ef80820f0a
Update 03174_least_greatest_ignore_null_input_values.reference 2024-11-28 11:51:06 +01:00
Robert Schulze
b772d0c9f4
Update 03174_least_greatest_ignore_null_input_values.sql 2024-11-28 11:50:07 +01:00
Robert Schulze
578e995786
Update 03174_least_greatest_ignore_null_input_values.sql 2024-11-28 11:48:43 +01:00
Robert Schulze
4c3d4043fa
Update 03174_least_greatest_ignore_null_input_values.sql 2024-11-28 11:47:55 +01:00
Sema Checherinda
a6cf336151 Merge branch 'master' into chesema-dtor-Finalizer 2024-11-28 11:23:31 +01:00
Antonio Andelic
3cc5c23647 Fix code 2024-11-28 11:11:05 +01:00
Antonio Andelic
4abe98ee62 Update NuRaft 2024-11-28 10:36:33 +01:00
Pablo Marcos
98ccca99f2 Add doc for SYSTEM LOAD PRIMARY KEY 2024-11-28 09:05:50 +00:00
Pablo Marcos
f1f5304484 Update test_grant_and_revoke/test_with_table_engine_grant.py::test_grant_all_on_table 2024-11-28 08:59:49 +00:00
Azat Khuzhin
d530468c59 Fix possible CI failures due to slow logs proxy tables shutdown 
This tables sends data to cloud from system.*_log tables, and
flush_on_detach will not help if the send is already in progress.

Let's disable distributed sends as a workaround for now, until the issue
will be fixed in a normal way.

CI: https://s3.amazonaws.com/clickhouse-test-reports/71406/9534a35696d8a82182d59ed24e4a604d7d11adcd/stress_test__asan_.html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-11-28 09:59:27 +01:00
Pablo Marcos
213275727f Keep loadPrimaryKeys single-threaded and simple 2024-11-28 08:39:24 +00:00
Igor Nikonov
97bd43894d Merge remote-tracking branch 'origin/master' into pr-skip-index-analysis-on-workers 2024-11-28 08:12:31 +00:00
nauu
a07111d30f update trusted contributors 2024-11-28 12:34:39 +08:00
kevinyhzou
a898a4e956 fix least/greatest 2024-11-28 12:27:21 +08:00
Jiebin Sun
e60b0b8508
Merge branch 'ClickHouse:master' into shared_mutex_optimization 2024-11-28 11:54:30 +08:00
zhanglistar
128080c249 delete toUnixTimestampEx 2024-11-28 11:42:59 +08:00