Commit Graph

96 Commits

Author SHA1 Message Date
vdimir
a228f7f419
[wip] ProfileCounters for each part 2023-02-14 14:11:09 +00:00
Alexander Tokmakov
70d1adfe4b
Better formatting for exception messages (#45449)
* save format string for NetException

* format exceptions

* format exceptions 2

* format exceptions 3

* format exceptions 4

* format exceptions 5

* format exceptions 6

* fix

* format exceptions 7

* format exceptions 8

* Update MergeTreeIndexGin.cpp

* Update AggregateFunctionMap.cpp

* Update AggregateFunctionMap.cpp

* fix
2023-01-24 00:13:58 +03:00
Zhiguo Zhou
be4d79e924 Deallocate memory of profile events out of critical section
To further shrink the critical section for releasing memory of the
profile events (ProfileEventsCountersAndMemory), this commit puts
the dealloaction out of the critical section while keeping the
memory move under lock. This change could mitigate the contention
for ThreadGroupStatus::mutex.
2023-01-11 09:58:30 +08:00
Raúl Marín
4d3217cb18
Merge branch 'master' into perf_experiment 2022-11-24 10:32:12 +01:00
Azat Khuzhin
ccde92f855 Fix incorrect UserTimeMicroseconds/SystemTimeMicroseconds accounting
After #40732 it became possible that getrusage() (from detachQuery(),
from buildPushingToViewsChain()) will be called for incorrect thread,
and so when the difference will be calculated it will be simply garbage.

But actually the root of this problem is #25714, after which it became
possible to have multiple ThreadStatus for one thread, and this is very
tricky (sigh).

Here are some other thoughts about it:
- Make ThreadStatus nested - decided that complexity does not worth it,
  at least only for this case
- Move some members into ThreadGroupStatus - will break per-thread
  statistics (and hence query_thread_log, BTW does somebody uses it?)
- Move some members into a separate structure

But decided to fix the issue w/o any refactoring, to make easy for
backport.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-11-18 21:10:43 +01:00
Raúl Marín
6e0a9452e7 Merge remote-tracking branch 'blessed/master' into perf_experiment 2022-10-25 15:25:06 +02:00
Azat Khuzhin
4e76629aaf Fixes for -Wshorten-64-to-32
- lots of static_cast
- add safe_cast
- types adjustments
  - config
  - IStorage::read/watch
  - ...
- some TODO's (to convert types in future)

P.S. That was quite a journey...

v2: fixes after rebase
v3: fix conflicts after #42308 merged
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-10-21 13:25:19 +02:00
Raúl Marín
227af9a074 Optimizations around ThreadStatus 2022-10-13 21:49:21 +02:00
Alexey Milovidov
ec5a32f534 Make thread_ids unique 2022-10-08 02:05:36 +02:00
Anton Popov
148995894f add system table asynchronous_inserts_log 2022-10-03 21:35:55 +00:00
Zhiguo Zhou
223c1230b6 Optimize the lock contentions for ThreadGroupStatus::mutex
The release of ThreadGroupStatus::finished_threads_counters_memory
via the getProfileEventsCountersAndMemoryForThreads method brings
lots of lock contentions for ThreadGroupStatus::mutex and lowers
the overall performance. This commit optimizes this performance
issue by replacing the method call with an equivalent but more
lightweight code block.
2022-09-22 16:23:16 +08:00
Frank Chen
f17d56b528 Merge branch 'master' into tracing_context_propagation 2022-08-30 14:24:36 +08:00
Azat Khuzhin
269453a646 Avoid leaking of ThreadGroupStatus::finished_threads_counters_memory
Cleanup them in ThreadStatus::detachQuery(), anyway they cannot be
received after by the client.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-08-29 11:08:50 +02:00
Frank Chen
99c37ce6c6
Merge branch 'master' into tracing_context_propagation 2022-08-25 10:07:16 +08:00
Alexey Milovidov
74e1f4dc61 Fix clang-tidy 2022-08-20 17:09:20 +02:00
Frank Chen
40c6e4c0d6 Merge remote-tracking branch 'origin/master' into tracing_context_propagation 2022-08-02 10:02:09 +08:00
Azat Khuzhin
b20407fab9 Fix NOEXCEPT_SCOPE (before it calls std::terminate and looses the exception)
Current implementation of NOEXCEPT_SCOPE will not work, you cannot
rethrow exception outside the catch block, this will simply terminate
(via std::terminate) the program.
In other words NOEXCEPT_SCOPE macro will simply call std::terminate on
exception and will lost original exception.

But if NOEXCEPT_SCOPE will accept the code that should be runned w/o
exceptions, then it can catch exception and log it, rewrite it in this
way.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-07-14 17:16:18 +03:00
Frank Chen
2e8c530bed Move thread trace context out of ThreadStatus 2022-07-07 17:41:10 +08:00
Yatsishin Ilya
b4e62e06e7 Use native Map 2022-07-04 14:07:29 +00:00
Alexander Tokmakov
39c0219c11 fixes 2022-06-16 19:41:32 +02:00
Robert Schulze
1a0b5f33b3
More consistent use of platform macros
cmake/target.cmake defines macros for the supported platforms, this
commit changes predefined system macros to our own macros.

__linux__ --> OS_LINUX
__APPLE__ --> OS_DARWIN
__FreeBSD__ --> OS_FREEBSD
2022-06-10 10:22:31 +02:00
Azat Khuzhin
16b1d2bee4 Remove outdated comment from ThreadStatusExt
Refs: https://github.com/ClickHouse/ClickHouse/pull/16121#discussion_r511500850

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-04-29 17:04:56 +03:00
Robert Schulze
b24ca8de52
Fix various clang-tidy warnings
When I tried to add cool new clang-tidy 14 warnings, I noticed that the
current clang-tidy settings already produce a ton of warnings. This
commit addresses many of these. Almost all of them were non-critical,
i.e. C vs. C++ style casts.
2022-04-20 10:29:05 +02:00
kssenii
36c583d0de Better version of cache on insert 2022-04-04 10:58:59 +02:00
Azat Khuzhin
cb70544dfe Move LockMemoryExceptionInThread and MemoryTrackerBlockerInThread 2022-01-10 22:39:10 +03:00
Azat Khuzhin
c1dea66907 Move TraceCollector into Interpreters
Since now it relies on SystemLog that is in Interpreters, and it cannot
be moved into Common, since it has lots of dependencies.
2022-01-10 22:35:42 +03:00
Azat Khuzhin
aee034a597 Use explicit template instantiation for SystemLog
- Move some code into module part to avoid dependency from IStorage in SystemLog
- Remove extra headers from SystemLog.h
- Rewrite some code that was relying on headers that was included by SystemLog.h

v2: rebase
v3: squash move into module part with explicit template instantiation
    (to make each commit self compilable after rebase)
2022-01-10 22:01:41 +03:00
Raúl Marín
3a7e524f9e Style 2021-12-09 18:24:28 +01:00
Raúl Marín
3fc4167c54 Rework how progress is reported in views 2021-12-09 17:08:29 +01:00
Azat Khuzhin
f0628073bb Use comments that can be parsed by clang-tidy in ThreadStatus::initQueryProfiler() 2021-11-26 01:13:55 +03:00
Azat Khuzhin
2840405c8a Fix typo in ThreadStatus::query_profiler_enabled 2021-11-26 01:13:55 +03:00
Azat Khuzhin
baf14444e6 Cleanup ProfileEvents and CurrentMetrics 2021-11-10 21:15:27 +03:00
Dmitry Novik
a33610aa32 Make read of Counters snapshot non-atomic 2021-10-12 13:46:32 +03:00
Dmitry Novik
356723427d WIP on ProfileEvents forwarding 2021-10-11 17:39:23 +03:00
Dmitry Novik
803b8623c1 Fix TCPHandler::sendProfileEvents 2021-10-11 17:29:50 +03:00
Dmitry Novik
e9b1e05461 Send profile events from all threads of current group 2021-10-11 17:29:50 +03:00
Nikolai Kochetov
340b53ef85 Remove some more streams. 2021-10-08 17:03:54 +03:00
Alexey Milovidov
fe6b7c77c7 Rename "common" to "base" 2021-10-02 10:13:14 +03:00
Nikolai Kochetov
02e5e662fd Refactor a little bit. 2021-09-21 13:35:41 +03:00
Alexey Milovidov
53ca1ebca6 Be in style 2021-08-25 03:58:49 +03:00
Raúl Marín
820a6e0987 Refactor MV stage metric setup and teardown 2021-08-10 10:49:12 +02:00
Raúl Marín
db0cdd696f Dependent views: Report progress 2021-07-27 16:29:34 +02:00
Raúl Marín
0d9bf0a08e Avoid havin multiple profilers in the same thread 2021-07-27 16:29:34 +02:00
Raúl Marín
14e8029e01 Initial basic tests and change name to not include uuid 2021-07-27 16:29:33 +02:00
Raúl Marín
eb994d8f04 Fix a couple of bugs
- Don't use threads if there is only one dependent view and parallel_view
- Reset the current_thread when destroying views ThreadStatus
2021-07-27 16:29:33 +02:00
Raúl Marín
7c1fcc94b0 View log: Make sure to only profile events during view execution 2021-07-27 16:29:33 +02:00
Raúl Marín
e0ee7839f7 Only use threads when necessary and log written status 2021-07-27 16:29:31 +02:00
Raúl Marín
6b9ec2a62e WIP 2021-07-27 16:28:28 +02:00
Raúl Marín
efe1938437 WIP: Slow improvements 2021-07-27 16:26:29 +02:00
Raúl Marín
ea5c02a605 WIP 2021-07-27 16:26:27 +02:00