Miсhael Stetsyuk
6d9a45eff1
Merge branch 'master' into startup-scripts-execution-state
2024-11-29 21:57:58 +00:00
Miсhael Stetsyuk
af04a3ce68
Merge pull request #72490 from ClickHouse/add-index-granularity-internal-array-size-metric
...
add metric MergeTreeIndexGranularityInternalArraysTotalSize
2024-11-29 19:17:03 +00:00
robot-clickhouse
fad3f2ff00
Automatic style fix
2024-11-29 17:54:14 +00:00
Michael Stetsyuk
b512997e50
better test
2024-11-29 17:45:55 +00:00
Michael Stetsyuk
997171a00e
add StartupScriptsExecutionState metric
2024-11-29 15:29:07 +00:00
Michael Stetsyuk
485f53e95e
add test
2024-11-29 14:42:36 +00:00
János Benjamin Antal
62368fb2a3
Merge pull request #72375 from jiebinn/shared_mutex_optimization
...
Optimize the mutex with shared_mutex in the memory tracker
2024-11-29 13:29:13 +00:00
Mikhail Artemenko
b81ee27750
Merge pull request #71406 from azat/automatic-external-aggregation
...
Automatic GROUP/ORDER BY to disk based on the memory usage
2024-11-29 12:27:43 +00:00
Anton Popov
938b7ad848
Merge pull request #72615 from CurtizJ/fix-system-load-primary-key
...
Fix tests for `SYSTEM LOAD PRIMARY KEY`
2024-11-29 12:22:06 +00:00
Alexey Katsman
d7167bd103
Merge pull request #72548 from alexkats/dict
...
Add source query validation for dictionaries
2024-11-29 11:25:40 +00:00
Robert Schulze
4c4c7b4fce
Update 03202_system_load_primary_key.sql
2024-11-29 12:08:57 +01:00
Daniil Ivanik
300d36bb72
Merge pull request #72277 from ClickHouse/divanik/iceberg_format_1_version_inconsistency_warning
...
Fix iceberg v1 format metadata json inconsistency
2024-11-29 10:49:15 +00:00
alesapin
d25b15a3b1
Merge pull request #72613 from ClickHouse/nopctest
...
Remove flaky test_page_cache
2024-11-29 10:16:13 +00:00
Sema Checherinda
54424ade94
Merge pull request #72395 from ClickHouse/chesema-dtor-Finalizer
...
make d-tor Finalizer more obvious
2024-11-29 08:05:14 +00:00
Nikita Taranov
523d3b4af7
Merge pull request #72555 from ClickHouse/raise_default_for_prealloc_in_ht_optimisation
...
Raise defaults for preallocation in HTs optimisation
2024-11-28 23:20:48 +00:00
Robert Schulze
4c36e8f427
Merge pull request #65519 from KevinyhZou/Fix_least_greast_diff
...
Make functions `least` and `greatest` ignore NULL arguments
2024-11-28 23:04:45 +00:00
Anton Popov
35133ebed9
fix tests for SYSTEM LOAD PRIMARY KEY
2024-11-28 22:58:45 +00:00
Michael Kolupaev
29b722df29
Remove flaky test_page_cache
2024-11-28 21:09:12 +00:00
Nikita Taranov
f00b0b02c9
Merge pull request #72566 from ClickHouse/fix_03167_improvement_table_name_too_long
...
Fix 03167_improvement_table_name_too_long
2024-11-28 17:02:26 +00:00
Vladimir Cherkasov
bc79653eed
Merge pull request #72586 from ClickHouse/vdimir/race_col_const_for_each_subsol
...
Fix data race in Squashing with ColumnConst
2024-11-28 16:21:47 +00:00
Pablo Marcos
d8a235b785
Merge pull request #67733 from Zawa-ll/66073-system-load-primary-key-rebase
...
Added statement `SYSTEM LOAD PRIMARY KEY`
2024-11-28 16:02:31 +00:00
Alex Katsman
23086e1424
Add source query validation for dictionaries
2024-11-28 15:38:07 +00:00
Nikita Taranov
f3f41d82df
Merge pull request #70598 from bigo-sg/more_jits
...
Enable JIT compilation for more expressions
2024-11-28 15:29:29 +00:00
Azat Khuzhin
dad435a10f
Ignore max_bytes_ratio_before_external_{order,group_by} if memory limit is not configured
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2024-11-28 16:10:04 +01:00
Azat Khuzhin
b0da170e62
Fix default value for group_by_overflow_mode
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2024-11-28 16:01:53 +01:00
Azat Khuzhin
a8eaf3ecde
Fix copy-paste typo for max_bytes_ratio_before_external_group_by doc
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2024-11-28 16:01:53 +01:00
Azat Khuzhin
bdb39a6fef
Change logic of automatic ORDER/GROUP BY to be based on available memory
...
- remove max_bytes_ratio_before_external_{order,group_by}_for_server
- change the way max_bytes_ratio_before_external_{order,group_by} works
Note, that it is not enough to transform ratio to bytes in
executeQuery(), since in this case it will not work for merges and
internal queries, plus, you have to reset them for Distributed engine
and update it for Merge/View/...
This patch also introduce some helpers (see MemoryTrackerUtils) and
adjust Aggregator::Params constructor to accept Settings object instead
of tons of arguments.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2024-11-28 16:01:53 +01:00
Azat Khuzhin
14e27b7de9
Move settings changes to 24.12
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2024-11-28 16:01:53 +01:00
Azat Khuzhin
ee271c2a06
Reduce copy-paste by introducing Aggregator::writeToTemporaryFileIfNeeded()
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2024-11-28 16:01:53 +01:00
Azat Khuzhin
520b21efa5
Respect max_bytes_ratio_before_external_sort_for_server in Aggregator::mergeOnBlock()
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2024-11-28 16:01:53 +01:00
Azat Khuzhin
2d3d0bb5cc
Respect max_bytes_ratio_before_external_sort_for_server for ORDER BY storage check
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2024-11-28 16:01:53 +01:00
Azat Khuzhin
2fab184ee4
Properly handle unlimited memory limit for automatic GROUP/ORDER BY
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2024-11-28 16:01:53 +01:00
Azat Khuzhin
db1f459cc4
tests: disable test_max_bytes_ratio_before_external_order_group_by_for_server under TSan
...
The problem is that TSan has some memory overhead which increases
process RSS to 10Gi in the middle of the query and it will fail.
Another option is to avoid syncing with RSS too frequently, but I doubt
that it is significant to run this test under TSan
v1: increase memory to 20Gi
Fixes: https://s3.amazonaws.com/clickhouse-test-reports/71406/b33be86dee3c7616e9193c339837b9e250810557/integration_tests__tsan__[6_6].html
v2: instead just disable under TSan
Fixes: https://s3.amazonaws.com/clickhouse-test-reports/71406/df68cf7362d825c3d83991fd74c391a49e73a8b1/integration_tests__tsan__[6_6].html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2024-11-28 16:01:53 +01:00
Azat Khuzhin
02aa6eac82
Cover max_bytes_ratio_before_external_{order,group_by}_for_server
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2024-11-28 16:01:53 +01:00
Azat Khuzhin
909eef401a
Implement max_bytes_ratio_before_external_sort_for_server
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2024-11-28 16:01:53 +01:00
Azat Khuzhin
d5a990346c
Implement max_bytes_ratio_before_external_group_by_for_server
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2024-11-28 16:01:53 +01:00
Azat Khuzhin
378b6c80a6
Automatic ORDER BY to disk (max_bytes_ratio_before_external_sort)
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2024-11-28 16:01:53 +01:00
Azat Khuzhin
a856c30c7c
Automatic GROUP BY to disk (max_bytes_ratio_before_external_group_by)
...
Simpler then max_bytes_before_external_group_by, that allows you not to
think about memory consumption of the query, and can be set globally to
i.e. 0.5 so that the external aggregation will be enabled once the
memory usage reaches the 50%.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2024-11-28 16:01:53 +01:00
Azat Khuzhin
c29907c009
Introduce getCurrentQueryMemoryUsage() helper for Aggregator
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2024-11-28 16:01:53 +01:00
Azat Khuzhin
1b27a17959
Add annotation for calling constructor of Aggregator::Params
...
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2024-11-28 16:01:53 +01:00
Vitaly Baranov
13d4c16a56
Merge pull request #72427 from vitlibar/parallel-creating-tables-while-restoring
...
Implement creating tables in parallel while restoring from a backup
2024-11-28 14:48:39 +00:00
Nikita Taranov
863f3fba79
Merge branch 'master' into raise_default_for_prealloc_in_ht_optimisation
2024-11-28 15:32:21 +01:00
Anton Popov
5b03379aba
Merge pull request #72479 from CurtizJ/randomize-primary-key-settings
...
Randomize settings for primary key cache
2024-11-28 14:30:03 +00:00
Alexey Milovidov
4bbadf64c6
Merge pull request #72109 from ClickHouse/pr-skip-index-analysis-on-workers
...
PR: do index analysis only on coordinator
2024-11-28 12:49:28 +00:00
Jiebin Sun
0bbe51d4e9
Merge branch 'ClickHouse:master' into shared_mutex_optimization
2024-11-28 20:48:21 +08:00
Alexey Milovidov
c8942bb213
Merge pull request #72558 from azat/ci/faster-shutdown
...
Fix possible CI failures due to slow logs proxy tables shutdown
2024-11-28 12:48:14 +00:00
Alexey Milovidov
17165c47c9
Merge pull request #72575 from nauu/update_trusted_contributors
...
Update trusted contributors
2024-11-28 12:32:40 +00:00
Vladimir Cherkasov
3dfa9648a7
Merge pull request #72574 from tbragin/patch-23
...
Update README.md - Update Meetups
2024-11-28 12:04:44 +00:00
vdimir
e875005567
Fix data race in Squashing with ColumnConst
2024-11-28 11:44:08 +00:00
Robert Schulze
e55705978d
Update LeastGreatestGeneric.h
2024-11-28 12:33:06 +01:00