thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-30 19:42:00 +00:00
Code Issues Packages Projects Releases Wiki Activity
89,852 Commits 9,190 Branches 1,782 Tags 2.2 GiB
615f37a488
Commit Graph

63 Commits

Author SHA1 Message Date
Dmitry Novik
a97ba9741e Fix possible deadlock in OvercommitTracker during logging 2022-05-17 18:07:52 +00:00
Dmitry Novik
2ed5a4013a
Revert "Revert "Memory overcommit: continue query execution if memory is available"" 2022-05-03 00:45:13 +02:00
alesapin
f0b7af0aa2
Revert "Memory overcommit: continue query execution if memory is available" 2022-05-03 00:36:50 +02:00
Dmitry Novik
71b6f89166
Merge pull request #35637 from ClickHouse/memory-overcommit-free 
Memory overcommit: continue query execution if memory is available
 2022-05-02 19:00:18 +02:00
Amos Bird
4a5e4274f0
base should not depend on Common 2022-04-29 10:26:35 +08:00
alesapin
04afe552f8 Trying to avoid segfaults 2022-04-25 22:51:09 +02:00
Dmitry Novik
63de3c9746 Add comment with explanation 2022-04-12 13:00:06 +00:00
Dmitry Novik
0d75e773ab Continue query execution if memory is freed 2022-03-26 18:25:58 +00:00
Dmitry Novik
80694006f2
Update src/Common/MemoryTracker.cpp 
Co-authored-by: alesapin <alesapin@clickhouse.com>
 2022-02-01 16:15:00 +03:00
Dmitry Novik
1310a89fb2
Update src/Common/MemoryTracker.cpp 
Co-authored-by: alesapin <alesapin@clickhouse.com>
 2022-02-01 16:14:48 +03:00
Dmitry Novik
4e612d0e1a
Merge branch 'master' into memory-overcommit 2022-01-18 23:33:29 +03:00
Dmitry Novik
83c663e2d6 Cleanup after code review 2022-01-18 12:21:59 +00:00
Azat Khuzhin
cb70544dfe Move LockMemoryExceptionInThread and MemoryTrackerBlockerInThread 2022-01-10 22:39:10 +03:00
Azat Khuzhin
c1dea66907 Move TraceCollector into Interpreters 
Since now it relies on SystemLog that is in Interpreters, and it cannot
be moved into Common, since it has lots of dependencies.
 2022-01-10 22:35:42 +03:00
Dmitry Novik
12101d82aa Fix overcommit ratio comparison and race condition 2021-12-06 21:34:52 +03:00
Dmitry Novik
9c7e6c7814 Use per user soft limit in GlobalOvercommitTracker 2021-11-09 16:02:36 +03:00
Dmitry Novik
0816dedaea Introduce memory overcommit 2021-11-09 16:02:36 +03:00
Azat Khuzhin
bf88f102fe Add ability to trace peak memory usage (with new trace_type - MemoryPeak) 2021-10-07 21:13:56 +03:00
Alexey Milovidov
fe6b7c77c7 Rename "common" to "base" 2021-10-02 10:13:14 +03:00
Sergei Semin
b9bdaf0e7b fix problem with MemoryTracker 2021-09-09 23:51:47 +03:00
tavplubix
bd3fe4fb41
Update MemoryTracker.cpp 2021-08-13 13:23:54 +03:00
benbiti
83ebcd45eb dup codes calc QueryMemoryLimitExceeded 2021-08-11 17:03:07 +08:00
Maksim Kita
8e192bef55 MemoryTracker enable throw logical error 2021-05-28 20:22:17 +03:00
Maksim Kita
fc5dbaa413 Updated MemoryTracker 2021-05-26 23:31:35 +03:00
Maksim Kita
60c20ed62d Update MemoryTracker 2021-05-25 23:54:48 +03:00
Maksim Kita
fadead3efe MemoryTracker new no throw 2021-05-25 22:23:26 +03:00
Alexey Milovidov
8f01af62d9 Merge branch 'master' into normalize-bigint 2021-04-25 06:57:44 +03:00
Azat Khuzhin
b6d7562cc2 Update comments for memoryTrackerCanThrow() 2021-04-14 23:38:41 +03:00
Azat Khuzhin
a46d65f99d Fix typo in comment for memoryTrackerCanThrow() 2021-02-05 01:29:46 +03:00
Alexey Milovidov
093108bf81 Normalize BigInt implementation 2021-01-27 03:54:57 +03:00
Alexey Milovidov
e482c5ea15 Fix TSan 2021-01-17 15:40:14 +03:00
Alexey Milovidov
950bfb3ec6 Allow change max_server_memory_usage without restart 2021-01-16 18:33:46 +03:00
Azat Khuzhin
61b2d0ce42 MemoryTracker: Do not ignore server memory limits during blocking by default 2021-01-15 22:46:58 +03:00
Alexander Tokmakov
ff6c3c75c2 add protection from unsafe allocations 2021-01-12 18:41:24 +03:00
Azat Khuzhin
780b86982a Respect memory tracker blocker level during deallocations 
Introduced-in: #17646
 2021-01-06 02:44:06 +03:00
Azat Khuzhin
a3ab429129 Split MemoryTracker header 
Allocator.h is included into almost everything (via PODArray.h), so keep
only small part of MemoryTracker.h for it.
 2020-12-25 23:49:22 +03:00
Azat Khuzhin
9d64ae2895 Extend BlockerInThread/LockExceptionInThread interface 
- Add level for BlockerInThread/LockExceptionInThread to make decisions
  based on the level
- Add block_fault_injections for LockExceptionInThread
 2020-12-25 23:49:22 +03:00
Azat Khuzhin
0bf6ed2e94 Make LockExceptionInThread/BlockerInThread nested 2020-12-01 10:34:25 +03:00
Azat Khuzhin
99c447d3f5 Add ability to block MEMORY_LIMIT_EXCEEDED exception explicitly 2020-11-28 11:15:26 +03:00
Azat Khuzhin
5088668736 Do not allow to throw MEMORY_LIMIT_EXCEEDED if there is uncaught exception 
Since this will lead to std::terminate
 2020-11-28 11:15:26 +03:00
Alexander Tokmakov
dbac7860b1 review suggestions 2020-11-11 16:27:54 +03:00
Alexander Tokmakov
b94cc5c4e5 remove more stringstreams 2020-11-10 21:22:26 +03:00
Alexey Milovidov
fd84d16387 Fix "server failed to start" error 2020-11-07 03:14:53 +03:00
Azat Khuzhin
114ea9b1eb Fix accounting for new/delete from different threads for VariableContext::Thread 
MemoryTracker assumes that for VariableContext::Thread new/delete may be
called from different threads, hence the amount of memory can go
negative.

However the MemoryTracker is nested, so even if the negative amount is
allowed for VariableContext::Thread it does not allowed for anything
upper, and hence the MemoryTracking will not be decremented properly.

Fix this, by passing initial size to the parent free.

This should fix memory drift for HTTP queries.
 2020-10-23 21:07:52 +03:00
Azat Khuzhin
72d7b6117e Use total_memory_tracker when there is no other MemoryTracker object. 
This should significantly reduce the MemoryTracking drift, test shows
that there is 0 drift after query storm (100 queries, via http/tcp/tcp
in one session).

TL;DR;

To track memory, clickhouse creates memory tracker object for each
thread **explicitly**, but until it is not created the memory
allocations are not under account.
There should not be lot of allocations w/o memory tracker, since most of
the time it is created early enough, but even this maybe enough to
trigger some problems.

Plus sometimes it is not possible to create it, for example some 3d
party library does not allow to do this explicitly:
- for example before #15740 allocations from librdkafka threads,
- or even worse, poco threads, they don't have any routines to do this.
This won't be a problem for `MemoryTracking` metric if the deallocation
will be done from the same thread w/o memory tracker (or vise versa),
but this is not always true.

NOTE, that this will slow down per-thread allocations w/o memory
tracker, since before this patch there were no memory tracking for them
while now they will be accounted in total_memory_tracker, and for
total_memory_tracker max_untracked_memory is always reached.
But this should not be significant.
 2020-10-23 21:07:52 +03:00
Alexey Milovidov
2619efadc8 Fix multiple issues with memory tracking 2020-10-21 03:31:34 +03:00
Peng Jian
9fa04cf48b Add QueryMemoryLimitExceeded event 2020-09-10 00:59:38 +08:00
alexey-milovidov
beed3c8244
Merge pull request #13500 from ClickHouse/hardening-and-better-error-messages 
Hardening and better error messages
 2020-08-08 17:13:43 +03:00
Alexey Milovidov
edd89a8610 Fix half of typos 2020-08-08 03:47:03 +03:00
Alexey Milovidov
b6625006dc Hardening and better error messages 2020-08-07 22:36:04 +03:00