thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 10:52:30 +00:00
Code Issues Packages Projects Releases Wiki Activity
108,361 Commits 9,328 Branches 1,783 Tags 2.1 GiB
d8cda3dbb8
Commit Graph

98 Commits

Author SHA1 Message Date
Dmitry Novik
ec1f6bfd37 Improve MEMERY_LIMIT_EXCEEDED exception message 2023-01-29 01:52:37 +00:00
Alexander Tokmakov
70d1adfe4b
Better formatting for exception messages (#45449) 
* save format string for NetException

* format exceptions

* format exceptions 2

* format exceptions 3

* format exceptions 4

* format exceptions 5

* format exceptions 6

* fix

* format exceptions 7

* format exceptions 8

* Update MergeTreeIndexGin.cpp

* Update AggregateFunctionMap.cpp

* Update AggregateFunctionMap.cpp

* fix
 2023-01-24 00:13:58 +03:00
Raúl Marín
948ce8db93 Revert "Merge pull request #38953 from ClickHouse/add-allocation-ptr-to-trace-log" 
This reverts commit e99849d031, reversing
changes made to 7ad3ff8a9e.
 2022-12-27 14:28:13 +01:00
Nikolai Kochetov
9a4b3493fd Merge branch 'master' into add-allocation-ptr-to-trace-log 2022-12-06 17:09:54 +01:00
Nikolai Kochetov
7649a3c4d7 Merge branch 'master' into add-allocation-ptr-to-trace-log 2022-12-01 12:12:23 +01:00
Anton Popov
312dd725b4 allow to send profile events to trace_log 2022-11-24 20:12:16 +00:00
Dmitry Novik
b17dc24a94 Do not reuse jemalloc memory in test 2022-11-21 16:10:47 +00:00
Azat Khuzhin
7b69a70e82 Fix frequent memory drift message and clarify things in comments 
Somethine like:

    2022.09.28 06:33:34.001433 [ 3133669 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 562.20 MiB, peak 562.21 MiB, will set to 562.20 MiB (RSS), difference: -70.46 MiB
    2022.09.28 06:33:35.001639 [ 3133669 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 562.20 MiB, peak 562.21 MiB, will set to 562.20 MiB (RSS), difference: -70.45 MiB

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-26 17:38:37 +02:00
Azat Khuzhin
f5f6f1b593 Add profile events for jemalloc purge 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-10-26 17:38:36 +02:00
Robert Schulze
fd86829824
Consolidate config_core.h into config.h 
Less duplication, less confusion ...
 2022-09-28 13:31:57 +00:00
Nikolai Kochetov
9e1bb19559
Purge jemalloc arenas in case of high memory usage. (#40277) 
* Purge jemalloc arenas in case of high memory usage.

* Purge jemalloc arenas in case of high memory usage.

* Get RSS before jemalloc counters. Try to avoid negative RSS.

* Try to avoid negative RSS.

* muzzy -> dirty

* Another fix.

* Update MemoryTracker.cpp

* Wait for purged memory.

* Revert "Wait for purged memory."

This reverts commit 53a2621a2d.
 2022-09-20 23:56:10 +02:00
Alexey Milovidov
fa62c7e982 Fix half of trash 2022-09-10 04:08:16 +02:00
Azat Khuzhin
28f2607d50 Fix rare race in updating peak of memory tracker 
Previously it was possible that allocation, that had been done with
locked MEMORY_LIMIT_EXCEEDED (LockMemoryExceptionInThread), will update
the peak to the amount that will include some other allocations.

Consider the following example (T1/T2 - different threads):

Initial state:

    MemoryTracker::amount = 0

Possible race:

    LockMemoryExceptionInThread locker;
T1: MemoryTracker::allocImpl(1MiB)      T2: MemoryTracker::allocImpl(1PiB)
                                            will_be = 1T, amount = 1T
    will_be = 1T+1MiB, amount=1T+1MiB
    peak = will_be
                                            amount = 0 // revert due to exceeding limit
                                            throw(MEMORY_LIMIT_EXCEEDED)

So as you can see T1 includes allocations from T2 into the peak.

Note, that almost every place that has memory_usage column uses
MemoryTracker::peak (not MemoryTracker::amount), this is true for
query_log (for part_log column name is peak_memory_usage, and obviously
it uses MemoryTracker::peak).

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-08-19 20:57:12 +02:00
Azat Khuzhin
ded70fbd53 Fix memory accounting in case of MEMORY_LIMIT_EXCEEDED errors 
In case of memory allocation had been failed, it should not be take into
account anywhere, otherwise it will report incorrect memory usage for
query and in logs.

I found this while was digging into OOM in stress tests, and saw memory
allocations that had been significantly greater the memory limit (even
with overcommit).

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-08-16 18:08:56 +02:00
Nikita Mikhaylov
2282f2f414
Merge branch 'master' into add-allocation-ptr-to-trace-log 2022-08-15 12:25:45 +02:00
Alexey Milovidov
216c1d8efa
Revert "Use separate counter for RSS in global memory tracker." 2022-08-14 09:50:12 +03:00
Nikita Mikhaylov
00d2c57a68
Merge branch 'master' into add-allocation-ptr-to-trace-log 2022-08-12 13:32:24 +02:00
Nikita Mikhaylov
8e8e845a89
Merge branch 'master' into separate-counter-for-rss-in-memory-tracker 2022-08-10 16:44:24 +02:00
Nikita Mikhaylov
f144eae388
Fix typo and extra dots in exception messages from OverCommitTracker (#39858) 2022-08-03 15:23:07 +02:00
Nikolai Kochetov
fa0707b4e7 Review fixes. 2022-07-26 17:51:09 +00:00
Nikolai Kochetov
fc6982f9e2 Fixing test. 2022-07-19 10:10:02 +00:00
Nikolai Kochetov
5a3cb07711 Fix rss 2022-07-18 21:24:41 +00:00
Nikolai Kochetov
3ee3144708 Merge branch 'master' into separate-counter-for-rss-in-memory-tracker 2022-07-18 23:20:38 +02:00
Nikolai Kochetov
87e5b31598 Do not include memory buffered by allocator into drift. 2022-07-18 21:16:16 +00:00
Nikolai Kochetov
8b56695073 Review fixes. 2022-07-08 14:46:11 +00:00
Nikolai Kochetov
c8b1d840ba Fixing build. 2022-07-07 15:49:15 +00:00
Nikolai Kochetov
0f231b8641 Merge branch 'master' into add-allocation-ptr-to-trace-log 2022-07-07 16:30:49 +02:00
Alexander Tokmakov
fdd2db4ace
Move check for denied allocations (#38858) 
* move check for denied allocations

* remove include

* make hardening softer
 2022-07-07 14:24:36 +02:00
Nikolai Kochetov
dcdf2e4527 Track allocation ptr in system.trace_log. Add aggregate function for flamegraph. 2022-07-07 11:54:41 +00:00
Alexey Milovidov
02adad3f27
Merge pull request #37860 from amosbird/remove-duplicated-log 
Remove duplicate peak mem log
 2022-07-07 08:16:06 +03:00
Nikolai Kochetov
b7529986c0 Use separate counter for RSS in global memory tracker. 2022-07-01 12:09:32 +00:00
Dmitry Novik
53d656d89f Fix deadlock in OvercommitTracker logging 2022-06-20 17:35:24 +00:00
Amos Bird
a5afb73c52
Remove duplicated peak mem log 
Also log "Processed in {} sec" a little bit earlier so that it's also
sent to client and have the log entry related to the query_id.
 2022-06-05 17:02:58 +08:00
Dmitry Novik
60b9d81773 Remove global_memory_usage_overcommit_max_wait_microseconds 2022-05-27 16:30:29 +00:00
Dmitry Novik
e9187ec4b7 Overcommit: update defaults, exception message and add ProfileEvent 2022-05-23 14:35:09 +00:00
Dmitry Novik
a97ba9741e Fix possible deadlock in OvercommitTracker during logging 2022-05-17 18:07:52 +00:00
Dmitry Novik
2ed5a4013a
Revert "Revert "Memory overcommit: continue query execution if memory is available"" 2022-05-03 00:45:13 +02:00
alesapin
f0b7af0aa2
Revert "Memory overcommit: continue query execution if memory is available" 2022-05-03 00:36:50 +02:00
Dmitry Novik
71b6f89166
Merge pull request #35637 from ClickHouse/memory-overcommit-free 
Memory overcommit: continue query execution if memory is available
 2022-05-02 19:00:18 +02:00
Amos Bird
4a5e4274f0
base should not depend on Common 2022-04-29 10:26:35 +08:00
alesapin
04afe552f8 Trying to avoid segfaults 2022-04-25 22:51:09 +02:00
Dmitry Novik
63de3c9746 Add comment with explanation 2022-04-12 13:00:06 +00:00
Dmitry Novik
0d75e773ab Continue query execution if memory is freed 2022-03-26 18:25:58 +00:00
Dmitry Novik
80694006f2
Update src/Common/MemoryTracker.cpp 
Co-authored-by: alesapin <alesapin@clickhouse.com>
 2022-02-01 16:15:00 +03:00
Dmitry Novik
1310a89fb2
Update src/Common/MemoryTracker.cpp 
Co-authored-by: alesapin <alesapin@clickhouse.com>
 2022-02-01 16:14:48 +03:00
Dmitry Novik
4e612d0e1a
Merge branch 'master' into memory-overcommit 2022-01-18 23:33:29 +03:00
Dmitry Novik
83c663e2d6 Cleanup after code review 2022-01-18 12:21:59 +00:00
Azat Khuzhin
cb70544dfe Move LockMemoryExceptionInThread and MemoryTrackerBlockerInThread 2022-01-10 22:39:10 +03:00
Azat Khuzhin
c1dea66907 Move TraceCollector into Interpreters 
Since now it relies on SystemLog that is in Interpreters, and it cannot
be moved into Common, since it has lots of dependencies.
 2022-01-10 22:35:42 +03:00
Dmitry Novik
12101d82aa Fix overcommit ratio comparison and race condition 2021-12-06 21:34:52 +03:00