thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 02:41:59 +00:00
Code Issues Packages Projects Releases Wiki Activity
102,045 Commits 9,328 Branches 1,783 Tags 2.1 GiB
77266ea754
Commit Graph

86 Commits

Author SHA1 Message Date
Dmitry Novik
b17dc24a94 Do not reuse jemalloc memory in test 2022-11-21 16:10:47 +00:00
Azat Khuzhin
7b69a70e82 Fix frequent memory drift message and clarify things in comments 
Somethine like:

    2022.09.28 06:33:34.001433 [ 3133669 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 562.20 MiB, peak 562.21 MiB, will set to 562.20 MiB (RSS), difference: -70.46 MiB
    2022.09.28 06:33:35.001639 [ 3133669 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 562.20 MiB, peak 562.21 MiB, will set to 562.20 MiB (RSS), difference: -70.45 MiB

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-26 17:38:37 +02:00
Azat Khuzhin
f5f6f1b593 Add profile events for jemalloc purge 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-26 17:38:36 +02:00
Robert Schulze
fd86829824
Consolidate config_core.h into config.h 
Less duplication, less confusion ...
 2022-09-28 13:31:57 +00:00
Nikolai Kochetov
9e1bb19559
Purge jemalloc arenas in case of high memory usage. (#40277) 
* Purge jemalloc arenas in case of high memory usage.

* Purge jemalloc arenas in case of high memory usage.

* Get RSS before jemalloc counters. Try to avoid negative RSS.

* Try to avoid negative RSS.

* muzzy -> dirty

* Another fix.

* Update MemoryTracker.cpp

* Wait for purged memory.

* Revert "Wait for purged memory."

This reverts commit 53a2621a2d.
 2022-09-20 23:56:10 +02:00
Alexey Milovidov
fa62c7e982 Fix half of trash 2022-09-10 04:08:16 +02:00
Azat Khuzhin
28f2607d50 Fix rare race in updating peak of memory tracker 
Previously it was possible that allocation, that had been done with
locked MEMORY_LIMIT_EXCEEDED (LockMemoryExceptionInThread), will update
the peak to the amount that will include some other allocations.

Consider the following example (T1/T2 - different threads):

Initial state:

    MemoryTracker::amount = 0

Possible race:

    LockMemoryExceptionInThread locker;
T1: MemoryTracker::allocImpl(1MiB)      T2: MemoryTracker::allocImpl(1PiB)
                                            will_be = 1T, amount = 1T
    will_be = 1T+1MiB, amount=1T+1MiB
    peak = will_be
                                            amount = 0 // revert due to exceeding limit
                                            throw(MEMORY_LIMIT_EXCEEDED)

So as you can see T1 includes allocations from T2 into the peak.

Note, that almost every place that has memory_usage column uses
MemoryTracker::peak (not MemoryTracker::amount), this is true for
query_log (for part_log column name is peak_memory_usage, and obviously
it uses MemoryTracker::peak).

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-08-19 20:57:12 +02:00
Azat Khuzhin
ded70fbd53 Fix memory accounting in case of MEMORY_LIMIT_EXCEEDED errors 
In case of memory allocation had been failed, it should not be take into
account anywhere, otherwise it will report incorrect memory usage for
query and in logs.

I found this while was digging into OOM in stress tests, and saw memory
allocations that had been significantly greater the memory limit (even
with overcommit).

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-08-16 18:08:56 +02:00
Alexey Milovidov
216c1d8efa
Revert "Use separate counter for RSS in global memory tracker." 2022-08-14 09:50:12 +03:00
Nikita Mikhaylov
8e8e845a89
Merge branch 'master' into separate-counter-for-rss-in-memory-tracker 2022-08-10 16:44:24 +02:00
Nikita Mikhaylov
f144eae388
Fix typo and extra dots in exception messages from OverCommitTracker (#39858) 2022-08-03 15:23:07 +02:00
Nikolai Kochetov
fa0707b4e7 Review fixes. 2022-07-26 17:51:09 +00:00
Nikolai Kochetov
fc6982f9e2 Fixing test. 2022-07-19 10:10:02 +00:00
Nikolai Kochetov
5a3cb07711 Fix rss 2022-07-18 21:24:41 +00:00
Nikolai Kochetov
3ee3144708 Merge branch 'master' into separate-counter-for-rss-in-memory-tracker 2022-07-18 23:20:38 +02:00
Nikolai Kochetov
87e5b31598 Do not include memory buffered by allocator into drift. 2022-07-18 21:16:16 +00:00
Alexander Tokmakov
fdd2db4ace
Move check for denied allocations (#38858) 
* move check for denied allocations

* remove include

* make hardening softer
 2022-07-07 14:24:36 +02:00
Alexey Milovidov
02adad3f27
Merge pull request #37860 from amosbird/remove-duplicated-log 
Remove duplicate peak mem log
 2022-07-07 08:16:06 +03:00
Nikolai Kochetov
b7529986c0 Use separate counter for RSS in global memory tracker. 2022-07-01 12:09:32 +00:00
Dmitry Novik
53d656d89f Fix deadlock in OvercommitTracker logging 2022-06-20 17:35:24 +00:00
Amos Bird
a5afb73c52
Remove duplicated peak mem log 
Also log "Processed in {} sec" a little bit earlier so that it's also
sent to client and have the log entry related to the query_id.
 2022-06-05 17:02:58 +08:00
Dmitry Novik
60b9d81773 Remove global_memory_usage_overcommit_max_wait_microseconds 2022-05-27 16:30:29 +00:00
Dmitry Novik
e9187ec4b7 Overcommit: update defaults, exception message and add ProfileEvent 2022-05-23 14:35:09 +00:00
Dmitry Novik
a97ba9741e Fix possible deadlock in OvercommitTracker during logging 2022-05-17 18:07:52 +00:00
Dmitry Novik
2ed5a4013a
Revert "Revert "Memory overcommit: continue query execution if memory is available"" 2022-05-03 00:45:13 +02:00
alesapin
f0b7af0aa2
Revert "Memory overcommit: continue query execution if memory is available" 2022-05-03 00:36:50 +02:00
Dmitry Novik
71b6f89166
Merge pull request #35637 from ClickHouse/memory-overcommit-free 
Memory overcommit: continue query execution if memory is available
 2022-05-02 19:00:18 +02:00
Amos Bird
4a5e4274f0
base should not depend on Common 2022-04-29 10:26:35 +08:00
alesapin
04afe552f8 Trying to avoid segfaults 2022-04-25 22:51:09 +02:00
Dmitry Novik
63de3c9746 Add comment with explanation 2022-04-12 13:00:06 +00:00
Dmitry Novik
0d75e773ab Continue query execution if memory is freed 2022-03-26 18:25:58 +00:00
Dmitry Novik
80694006f2
Update src/Common/MemoryTracker.cpp 
Co-authored-by: alesapin <alesapin@clickhouse.com>
 2022-02-01 16:15:00 +03:00
Dmitry Novik
1310a89fb2
Update src/Common/MemoryTracker.cpp 
Co-authored-by: alesapin <alesapin@clickhouse.com>
 2022-02-01 16:14:48 +03:00
Dmitry Novik
4e612d0e1a
Merge branch 'master' into memory-overcommit 2022-01-18 23:33:29 +03:00
Dmitry Novik
83c663e2d6 Cleanup after code review 2022-01-18 12:21:59 +00:00
Azat Khuzhin
cb70544dfe Move LockMemoryExceptionInThread and MemoryTrackerBlockerInThread 2022-01-10 22:39:10 +03:00
Azat Khuzhin
c1dea66907 Move TraceCollector into Interpreters 
Since now it relies on SystemLog that is in Interpreters, and it cannot
be moved into Common, since it has lots of dependencies.
 2022-01-10 22:35:42 +03:00
Dmitry Novik
12101d82aa Fix overcommit ratio comparison and race condition 2021-12-06 21:34:52 +03:00
Dmitry Novik
9c7e6c7814 Use per user soft limit in GlobalOvercommitTracker 2021-11-09 16:02:36 +03:00
Dmitry Novik
0816dedaea Introduce memory overcommit 2021-11-09 16:02:36 +03:00
Azat Khuzhin
bf88f102fe Add ability to trace peak memory usage (with new trace_type - MemoryPeak) 2021-10-07 21:13:56 +03:00
Alexey Milovidov
fe6b7c77c7 Rename "common" to "base" 2021-10-02 10:13:14 +03:00
Sergei Semin
b9bdaf0e7b fix problem with MemoryTracker 2021-09-09 23:51:47 +03:00
tavplubix
bd3fe4fb41
Update MemoryTracker.cpp 2021-08-13 13:23:54 +03:00
benbiti
83ebcd45eb dup codes calc QueryMemoryLimitExceeded 2021-08-11 17:03:07 +08:00
Maksim Kita
8e192bef55 MemoryTracker enable throw logical error 2021-05-28 20:22:17 +03:00
Maksim Kita
fc5dbaa413 Updated MemoryTracker 2021-05-26 23:31:35 +03:00
Maksim Kita
60c20ed62d Update MemoryTracker 2021-05-25 23:54:48 +03:00
Maksim Kita
fadead3efe MemoryTracker new no throw 2021-05-25 22:23:26 +03:00
Alexey Milovidov
8f01af62d9 Merge branch 'master' into normalize-bigint 2021-04-25 06:57:44 +03:00