thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-24 08:32:02 +00:00
Code Issues Packages Projects Releases Wiki Activity
101,684 Commits 9,168 Branches 1,781 Tags 2.1 GiB
2cfbc27f13
Commit Graph

101684 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Azat Khuzhin
2cfbc27f13 Fix incorrect memory accounting because of merges/mutations 
In case of merges/mutations, separate MemoryTracker object is used, to
track memory separately, to report this metrics.

However, the tail of those allocations hadn't been taken into account,
and in case of lots of mutations/merges it can be significant.

100% reproducible with the following example:

Start the server w/o frequent sync with RSS:

    $ clickhouse-server -C /src/programs/server/config.xml -- --asynchronous_metrics_update_period_s=86400

Run the following snippet:

    create table data (key Int, value Nullable(Decimal(38, 0))) engine=ReplicatedMergeTree('/ch/data', '1') order by key
    insert into data values (1, -13028236692093846346337460743176821145)
    alter table data modify column value UInt256
    Code: 341. DB::Exception: Received from localhost:9000. DB::Exception: Exception happened during execution of mutation 'mutation_2.txt' with part 'all_1_1_0' reason: 'Code: 407. DB::Exception: Convert overflow: while executing 'FUNCTION _CAST(value :: 1, 'UInt256' :: 2) -> _CAST(value, 'UInt256') UInt256 : 3': (while reading from part ./store/dd9/dd9ef183-1270-4877-af22-8a000bdcc1b8/all_1_1_0/): While executing MergeTreeInOrder. (DECIMAL_OVERFLOW) (version 22.12.1.1)'. This error maybe retryable or not. In case of unretryable error, mutation can be killed with KILL MUTATION query. (UNFINISHED)

And track the memory:

    select value/1e9 from system.metrics where metric like 'MemoryTracking'

Reported-by: @filimonov
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-22 19:40:35 +01:00
Azat Khuzhin
5da2f52722 Use Int64 over UInt64 for prev_untracked_memory* in MemoryTrackerThreadSwitcher 
Since those types originally is Int64

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-22 19:40:35 +01:00
Azat Khuzhin
05f30aaad4 Fix logger name for MutateFromLogEntryTask 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-11-22 19:40:35 +01:00
Alexey Milovidov
b6fd28cd85
Merge pull request #43409 from azat/local/fix-uaf 
Fix possible heap-use-after-free in local if history file cannot be created
 2022-11-22 19:27:08 +01:00
Alexey Milovidov
c23b8c69db
Merge pull request #42791 from azat/fix-pe-overflow 
Fix incorrect UserTimeMicroseconds/SystemTimeMicroseconds accounting
 2022-11-22 19:22:51 +01:00
Alexey Milovidov
75dbd6828a
Merge pull request #43082 from Algunenano/werror_base 
Add global flags to base/ libraries
 2022-11-22 19:18:44 +01:00
Alexey Milovidov
f42be5ffd3
Merge pull request #43275 from azat/build/sanitize-targets 
Sanitize thirdparty libraries for public flags
 2022-11-22 19:18:26 +01:00
Alexey Milovidov
7dae1b08c6
Merge pull request #43448 from ClickHouse/modify-test 
Modify test slightly
 2022-11-22 18:31:00 +01:00
Alexey Milovidov
0165f56a5c
Merge pull request #43435 from ClickHouse/fix-flaky-test-3 
Fix flaky test
 2022-11-22 18:30:39 +01:00
Alexey Milovidov
c8b068b4ff
Merge pull request #43447 from ClickHouse/netlink-remove-noise 
Remove noise from logs about NetLink in Docker
 2022-11-22 18:29:43 +01:00
Alexey Milovidov
ae8391c5dc
Merge pull request #43436 from ClickHouse/fix-bc-check 
Fix backward compatibility check
 2022-11-22 18:08:59 +01:00
Alexander Gololobov
c636c323ea
Merge pull request #43400 from socketpair/sd_notify 
Fix #8685 - added systemd sd_notify implementation
 2022-11-22 18:07:50 +01:00
Kseniia Sumarokova
cef9e5108e
Merge pull request #43325 from kssenii/named-collections-access-fix 
Restrict default access to show named collections
 2022-11-22 16:51:17 +01:00
Alexander Gololobov
32d66659e5
Merge pull request #43364 from azat/server-stacktrace-on-fail 
Provide full stacktrace in case of uncaught exception during server startup
 2022-11-22 15:07:57 +01:00
Sergei Trifonov
1f615e4588
Merge pull request #42660 from ClickHouse/storage-system-moves 
Add active moves system table
 2022-11-22 15:07:22 +01:00
Sergei Trifonov
59f422f10c
Merge branch 'master' into storage-system-moves 2022-11-22 14:46:00 +01:00
Kruglov Pavel
98d6b96c82
Merge pull request #42033 from mark-polokhov/BSONEachRow 
Add BSONEachRow input/output format
 2022-11-22 14:45:21 +01:00
Kseniia Sumarokova
5c90d5aa7e
Merge pull request #43253 from xiedeyantu/fix-s3-glob 
fix s3 support question mark wildcard
 2022-11-22 14:26:56 +01:00
Kseniia Sumarokova
392963012a
Merge pull request #43404 from azat/part.log.uuid 
Add table_uuid to system.parts
 2022-11-22 14:25:34 +01:00
Kruglov Pavel
62bfb5547c
Merge pull request #43368 from Avogar/fix-if-nothing 
Fix possible `Cannot create non-empty column with type Nothing` in functions if/multiIf
 2022-11-22 14:11:13 +01:00
Sergei Trifonov
3177ed050f
Merge pull request #43307 from ClickHouse/cpu-progress-protocol-fix 
fix CPU load indication on client
 2022-11-22 14:09:55 +01:00
Alexander Gololobov
31bffdd735
Merge pull request #43460 from ClickHouse/smaller_buffer_for_small_files 
Use smaller buffer for small files
 2022-11-22 14:00:22 +01:00
Anton Popov
2572202a16
Merge pull request #43461 from CurtizJ/merging-42064 
Merging #42064
 2022-11-22 13:55:01 +01:00
Mikhail f. Shiryaev
8ce5d90c91
Merge pull request #43467 from ClickHouse/use-bots-tokens 
Use all parameters with prefixes from ssm
 2022-11-22 13:46:27 +01:00
Mikhail f. Shiryaev
7048f36fc7
Save unnecessary API requests 2022-11-22 13:37:08 +01:00
Dmitry Novik
e67bda65a0
Merge pull request #43453 from ClickHouse/fix-hard-limit-in-test 
Do not reuse jemalloc memory in test_global_overcommit
 2022-11-22 13:24:34 +01:00
Mikhail f. Shiryaev
fb95ca43d8
Use all parameters with prefixes from ssm 2022-11-22 12:43:17 +01:00
Alexander Tokmakov
7417d8b7c0
Merge pull request #43038 from Algunenano/revert_SingleValueDataString 
Fix regression on SingleValueDataString
 2022-11-22 14:28:30 +03:00
Alexander Tokmakov
cab33c02af
Merge branch 'master' into revert_SingleValueDataString 2022-11-22 14:27:58 +03:00
Antonio Andelic
843401c3a3
Merge pull request #43323 from ClickHouse/fix-describe-delta-lake-hudi 
Fix `DESCRIBE` for `deltaLake` and `hudi` table functions
 2022-11-22 09:09:39 +01:00
Anton Popov
4502fc0e81 fix flaky tests 2022-11-22 01:03:21 +00:00
Anton Popov
0479d8d9f5
Merge pull request #43387 from ClickHouse/fix_default_value_used_in_row_level_filter 
Fix default value used in row level filter
 2022-11-22 01:46:45 +01:00
Vitaly Baranov
6c2b53f0ae
Merge pull request #42940 from filimonov/exceptions_with_masker 
Add SensitiveDataMasker to exceptions messages
 2022-11-21 23:20:15 +01:00
Alexander Gololobov
5afd2a1add Pass file size to better choose buffer size 2022-11-21 22:36:45 +01:00
Alexander Gololobov
86a0dd010f Use read_hint and file_size for choosing buffer size 2022-11-21 22:36:28 +01:00
kssenii
fe6c36f429 Update .reference 2022-11-21 21:36:31 +01:00
kssenii
eb23e268ee Merge remote-tracking branch 'upstream/master' into named-collections-access-fix 2022-11-21 21:26:18 +01:00
Sergei Trifonov
94a7614c45
Merge branch 'master' into storage-system-moves 2022-11-21 20:10:12 +01:00
Anton Popov
5532e3db95 Merge branch 'filesystem-function' of https://github.com/ucasfl/ClickHouse into merging-42064 2022-11-21 18:23:11 +00:00
Kruglov Pavel
581e57be9f
Merge pull request #43236 from xiedeyantu/fix_hdfscluster_connection_fail 
fix skip_unavailable_shards does not work using hdfsCluster table function
 2022-11-21 19:22:16 +01:00
Igor Nikonov
47ce029cbf
Merge pull request #43326 from ClickHouse/igor/test_read_only_fast_fix 
Fix: make test_read_only_table more stable
 2022-11-21 18:25:14 +01:00
serxa
faef2e9b00 fix moves list entry creation 2022-11-21 17:08:54 +00:00
chen
96543074fe
Update S3Common.cpp 2022-11-22 01:03:31 +08:00
serxa
c425da06b2 fix column name 2022-11-21 16:16:53 +00:00
Dmitry Novik
b17dc24a94 Do not reuse jemalloc memory in test 2022-11-21 16:10:47 +00:00
Nikolay Degterinsky
65b5523bdb
Merge pull request #43350 from evillique/fix-parser-5 
Fix function parameters parsing
 2022-11-21 16:23:02 +01:00
Alexander Gololobov
63196a1e9e Close the socket; some cleanups 2022-11-21 16:05:47 +01:00
Alexey Milovidov
3e72acbb85 Modify test slightly 2022-11-21 15:18:02 +01:00
Alexey Milovidov
4ab6896ece Remove noise from logs about NetLink in Docker 2022-11-21 15:16:27 +01:00
Kruglov Pavel
406c50277b
Merge pull request #42779 from Avogar/add-format-func-doc 
Add documentation for 'format' table function
 2022-11-21 15:16:26 +01:00