Commit Graph

55 Commits

Author SHA1 Message Date
Alexey Milovidov
dbca269ec7
Merge pull request #40321 from azat/mem/untracked_memory_limit_increase
Remove ThreadStatus::untracked_memory_limit_increase
2022-09-04 04:42:24 +03:00
Azat Khuzhin
ec2e671d20 Remove ThreadStatus::untracked_memory_limit_increase
It looks useless nowadays, because operator new cannot throw
MEMORY_LIMIT_EXCEEDED today, and so any code that works on Exception is
likely safe.

Refs: #40249
Refs: #24483
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-08-22 13:26:18 +03:00
Frank Chen
a3b6ad2a65
Merge branch 'master' into tracing_context_propagation 2022-08-18 20:59:07 +08:00
Nikolai Kochetov
8e4c967fc9 Try to account untracked memory (per thread) more correctly. 2022-08-16 18:09:05 +02:00
Raúl Marín
11a274e990 Clean up constinit usage and add a comment 2022-08-11 13:27:53 +02:00
Raúl Marín
d7a545e30d Try to optimize CurrentMemoryTracker alloc and free 2022-08-10 22:05:09 +02:00
Frank Chen
57dde8c250 Clean up header inclusion
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
2022-08-02 15:43:19 +08:00
Frank Chen
40c6e4c0d6 Merge remote-tracking branch 'origin/master' into tracing_context_propagation 2022-08-02 10:02:09 +08:00
Robert Schulze
13482af4ee
First try at reducing the use of StringRef
- to be replaced by std::string_view
- suggested in #39262
2022-07-17 17:26:02 +00:00
Frank Chen
57a7e4a7c9 Remove old API reference 2022-07-07 17:42:35 +08:00
Frank Chen
2e8c530bed Move thread trace context out of ThreadStatus 2022-07-07 17:41:10 +08:00
Azat Khuzhin
c29768325c Print query in one line on fatal errors
executeQuery() logging query without new lines in SQL, and it is useful,
since you can grep and see the whole query (usually).

So let's use the same in fatal error handler to make CI reports more
informative.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-06-03 18:55:53 +03:00
kssenii
f0e0d97752 Fix lask of query id in merges 2022-03-29 19:49:42 +02:00
Azat Khuzhin
6aebc3e94c Do not loose ProfileEvents in case of thread destroyed before
v2: drop std::move and add copy ctor for ProfileEvents::Counter::Snapshot
v2: remove std::move
2021-12-19 23:43:05 +03:00
Azat Khuzhin
ee0258e128 Add sanity check into ThreadStatus::disableProfiling() 2021-11-26 01:13:55 +03:00
Azat Khuzhin
2840405c8a Fix typo in ThreadStatus::query_profiler_enabled 2021-11-26 01:13:55 +03:00
Azat Khuzhin
8a209a78d7 Set query_id for mutations/merges
This will allow to distinguish allocations in trace_log.
2021-10-15 01:43:28 +03:00
Dmitry Novik
356723427d WIP on ProfileEvents forwarding 2021-10-11 17:39:23 +03:00
Dmitry Novik
e9b1e05461 Send profile events from all threads of current group 2021-10-11 17:29:50 +03:00
Alexey Milovidov
fe6b7c77c7 Rename "common" to "base" 2021-10-02 10:13:14 +03:00
Nikolai Kochetov
f569a3e3f7 Merge branch 'master' into rewrite-pushing-to-views 2021-09-09 20:30:23 +03:00
Nikolai Kochetov
4d821efa15 Rewrite PushingToViewsBlockOutputStream part 2. 2021-08-26 11:01:26 +03:00
Alexey Milovidov
53ca1ebca6 Be in style 2021-08-25 03:58:49 +03:00
Nikolai Kochetov
6b1030c9b8 Rewrite PushingToViewsBIS part 1. 2021-08-13 19:32:29 +03:00
Raúl Marín
820a6e0987 Refactor MV stage metric setup and teardown 2021-08-10 10:49:12 +02:00
Raúl Marín
0d9bf0a08e Avoid havin multiple profilers in the same thread 2021-07-27 16:29:34 +02:00
Raúl Marín
7c1fcc94b0 View log: Make sure to only profile events during view execution 2021-07-27 16:29:33 +02:00
Raúl Marín
6b9ec2a62e WIP 2021-07-27 16:28:28 +02:00
Raúl Marín
ea5c02a605 WIP 2021-07-27 16:26:27 +02:00
Alexey Milovidov
e905883c75 More fixes for PVS-Studio 2021-05-08 19:12:31 +03:00
Ivan
495c6e03aa
Replace all Context references with std::weak_ptr (#22297)
* Replace all Context references with std::weak_ptr

* Fix shared context captured by value

* Fix build

* Fix Context with named sessions

* Fix copy context

* Fix gcc build

* Merge with master and fix build

* Fix gcc-9 build
2021-04-11 02:33:54 +03:00
Alexander Tokmakov
ffaa8e34a6 minor code improvements around ThreadStatus 2021-01-28 16:57:36 +03:00
kssenii
c1702f34ee Add factories info into system.query_log 2021-01-21 15:46:37 +00:00
Amos Bird
272311f6bd
Extensive information in system.query_log 2020-12-18 14:13:58 +08:00
Alexander Kuzmenkov
f326536ef0 fixup 2020-11-19 15:30:44 +03:00
Alexander Kuzmenkov
1570320e20 fixes for context hierarchy 2020-11-18 20:43:18 +03:00
Alexander Kuzmenkov
0530c40cd8 fixes 2020-11-10 08:50:32 +03:00
Alexander Kuzmenkov
26229ed231 tmp spans for threads
(doesn't compile because of json metadata changes)
2020-11-09 18:07:38 +03:00
Azat Khuzhin
3be8a56f5c Add log_queries_min_query_duration_ms
Only queries slower then the value of this setting will go to system.query_log,
i.e. something like slow_query_log in mysql.

v2: log_queries_min_time renamed to log_queries_min_query_duration_ms
v3: add current_database into system.query_thread_log
v4: rewrite test using current_database
v5: fix query_duration_ms in system.query_thread_log
2020-11-02 21:34:54 +03:00
Azat Khuzhin
5bab7a5bc7 Add current_database into query_thread_log 2020-10-30 21:16:10 +03:00
Azat Khuzhin
72d7b6117e Use total_memory_tracker when there is no other MemoryTracker object.
This should significantly reduce the MemoryTracking drift, test shows
that there is 0 drift after query storm (100 queries, via http/tcp/tcp
in one session).

TL;DR;

To track memory, clickhouse creates memory tracker object for each
thread **explicitly**, but until it is not created the memory
allocations are not under account.
There should not be lot of allocations w/o memory tracker, since most of
the time it is created early enough, but even this maybe enough to
trigger some problems.

Plus sometimes it is not possible to create it, for example some 3d
party library does not allow to do this explicitly:
- for example before #15740 allocations from librdkafka threads,
- or even worse, poco threads, they don't have any routines to do this.
This won't be a problem for `MemoryTracking` metric if the deallocation
will be done from the same thread w/o memory tracker (or vise versa),
but this is not always true.

NOTE, that this will slow down per-thread allocations w/o memory
tracker, since before this patch there were no memory tracking for them
while now they will be accounted in total_memory_tracker, and for
total_memory_tracker max_untracked_memory is always reached.
But this should not be significant.
2020-10-23 21:07:52 +03:00
Azat Khuzhin
082f30a3a1 Introduce ThreadStatus::applyQuerySettings() to reduce copy-paste 2020-10-17 15:01:53 +03:00
bharatnc
44e168bea7 query_start_time_microseconds field in system.query_log
Add a new field called `query_start_time_microseconds`  that will add
`microseconds`. This new field will be of type `DateTime64`.
2020-09-04 09:42:18 -07:00
Vitaly Baranov
18b21511a9
Merge pull request #13013 from vitlibar/implement-custom-settings
Implement custom settings
2020-08-02 05:01:14 +03:00
Vitaly Baranov
18e3f1f60d Split SettingsCollection.h into 3 files: SettingsFields.h, SettingsEnums.h, SettingsCollection.h 2020-07-31 19:11:27 +03:00
Alexey Milovidov
31cbdd1a56 system.crash_log: development 2020-07-09 07:15:45 +03:00
Alexey Milovidov
7ba25bd799 Make it work 2020-06-20 14:17:15 +03:00
Alexander Kuzmenkov
6e3bbf83e3 thread-local perf events 2020-05-29 16:04:32 +03:00
Andrey Skobtsov
cae41826a8 Merge branch 'master' into perf_events_metrics
# Conflicts:
#	src/Common/ThreadProfileEvents.cpp
#	src/Common/ThreadProfileEvents.h
#	src/Common/ThreadStatus.cpp
#	src/Common/ThreadStatus.h
2020-05-17 00:21:22 +03:00
Alexey Milovidov
600b396f1b Merge remote-tracking branch 'origin/master' into sampling-memory-profiler 2020-05-01 21:48:20 +03:00