thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-16 03:12:43 +00:00
Code Issues Packages Projects Releases Wiki Activity
115,637 Commits 9,326 Branches 1,783 Tags 2.1 GiB
092b06f32e
Commit Graph

5763 Commits

Author SHA1 Message Date
serxa
23defcbf6a log progress in AsyncLoader 2023-05-01 12:08:50 +00:00
Sergei Trifonov
b390a07698
Merge branch 'master' into async-loader 2023-05-01 14:04:07 +02:00
Nikolay Degterinsky
f64ef88197 Merge remote-tracking branch 'upstream/master' into bcrypt 2023-04-30 21:39:16 +00:00
alesapin
ec2c860a39 Merge branch 'fix_zero_copy_not_atomic' of github.com:ClickHouse/ClickHouse into fix_zero_copy_not_atomic 2023-04-30 15:02:27 +02:00
alesapin
daa0b095ed Merge branch 'master' into fix_zero_copy_not_atomic 2023-04-29 21:26:26 +02:00
Kseniia Sumarokova
d4aa96e262
Merge branch 'master' into better-local-object-storage 2023-04-28 21:05:05 +02:00
Robert Schulze
7609832606
Unpoison stack frame ptrs from libunwind for msan 
Turned out that in msan builds, stack frame pointers explicitly returned
by unw_backtrace() in Common/StackTrace.cpp were unpoisened, but stack
frame pointers returned from the same function under the
STD_EXCEPTION_HAS_STACK_TRACE define in std::exception (see (*)) were
not. I tried unpoisoning directly in libcxx. Since this didn't work,
this PR unpoisons where get_stack_trace_frames() is called.

I don't know why this issue never occurred in CI but it fixes #48385 for
me (where msan complains about reads of uninitialized data when query
cache squashes and compresses chunks produced from SELECT last_error_trace
FROM system.errors).

Fixes: #48385

(*) https://github.com/ClickHouse/llvm-project/pull/7/files#diff-8c81d74ef72967be2fa27a31275000aa18532a583490f9673c62d7b981a5b39c
 2023-04-28 17:43:12 +00:00
Alexey Milovidov
cd171bef7d
Merge pull request #45999 from sichenzhao/cgroup_v2 
Updates AsyncMetrics cgroup-memory to support cgroupv2
 2023-04-28 20:39:53 +03:00
alesapin
70ee02a3eb Add node holder 2023-04-28 17:39:32 +02:00
Alexey Milovidov
c86d3daee5
Update AsynchronousMetrics.cpp 2023-04-28 16:21:37 +03:00
alesapin
ef49603432 Merge branch 'master' into fix_zero_copy_not_atomic 2023-04-28 14:42:30 +02:00
kssenii
298ed454ce Merge remote-tracking branch 'upstream/master' into better-local-object-storage 2023-04-28 13:26:55 +02:00
Alexey Milovidov
4dccd5f9b0
Update AsynchronousMetrics.cpp 2023-04-27 23:20:38 +03:00
Nikolay Degterinsky
df1e56b985 Merge remote-tracking branch 'upstream/master' into bcrypt 2023-04-27 17:01:58 +00:00
Robert Schulze
b6e65ddeff
Merge branch 'master' into add-examples-to-functions 2023-04-27 16:11:32 +02:00
Kseniia Sumarokova
45e2d296f9
Merge branch 'master' into better-local-object-storage 2023-04-27 14:54:04 +02:00
Alexey Milovidov
c5793f9684
Fix build 2023-04-27 14:56:51 +03:00
Sergei Trifonov
85d7816d65
Update src/Common/AsyncLoader.h 2023-04-27 13:55:26 +02:00
Robert Schulze
096667be45
More fixes 2023-04-27 09:33:52 +00:00
Alexey Milovidov
1bb685a081
Fix error. 2023-04-27 01:32:54 +03:00
Robert Schulze
739bfc9b8d
Fix categories 2023-04-26 21:23:26 +00:00
Robert Schulze
d72030b876
Various fixes 2023-04-26 20:38:41 +00:00
serxa
94f730ded0 Merge branch 'async-loader' of github.com:ClickHouse/ClickHouse into async-loader 2023-04-26 18:40:26 +00:00
serxa
173a8a3795 fix tidy build 2023-04-26 18:39:51 +00:00
alesapin
f3d1d3b58b Merge branch 'master' into fix_zero_copy_not_atomic 2023-04-26 18:00:31 +02:00
Sergei Trifonov
bb4109b3e1
Merge branch 'master' into async-loader 2023-04-26 15:00:59 +02:00
Alexey Milovidov
88ce78a7be
Merge branch 'master' into cgroup_v2 2023-04-26 13:55:12 +03:00
alesapin
e5f71c3b74 Fix zero copy not atomic 2023-04-26 00:25:04 +02:00
alesapin
3a24f0b6c6 fxi 2023-04-25 15:33:41 +02:00
Anton Popov
cc5acfbe62 add profile event 2023-04-25 01:37:37 +00:00
Alexey Milovidov
dee0b78303
Merge pull request #48745 from Algunenano/arenita 
Don't count unused arena bytes as read_bytes in AggregateFunction types
 2023-04-25 00:53:11 +03:00
Alexey Milovidov
761ed890a7
Update AsynchronousMetrics.cpp 2023-04-25 00:45:57 +03:00
Anton Popov
38622d0770 add settings to delay or throw in case of too many mutations 2023-04-24 18:21:49 +00:00
alesapin
d28b71e69d
Merge pull request #49078 from ClickHouse/fix-false-report 
Prevent false positive report by static analyzer
 2023-04-24 13:08:37 +02:00
Robert Schulze
ded8eca041
Merge pull request #48284 from ClickHouse/rs/qc-quota 
Query Cache: Allow per-user quotas
 2023-04-24 11:32:32 +02:00
Vladimir C
6090648fc8
Merge pull request #43606 from arthurpassos/parsey_key_value_function 2023-04-24 11:03:28 +02:00
Raúl Marín
868b4e41fd Arena used bytes 2023-04-24 10:46:23 +02:00
Raúl Marín
f0e045bb3d Merge remote-tracking branch 'blessed/master' into arenita 2023-04-24 10:42:56 +02:00
Alexey Milovidov
ff5ec30b0b
Merge pull request #49005 from ClickHouse/fix-ibm 
Reimplement #48986
 2023-04-24 01:39:22 +03:00
Alexey Milovidov
f4bd5a0667 Prevent false positive report by static analyzer 2023-04-23 22:45:12 +02:00
Alexey Milovidov
c3749d652e
Update AsynchronousMetrics.cpp 2023-04-23 23:24:57 +03:00
Alexey Milovidov
40bae4b611
Merge branch 'master' into cgroup_v2 2023-04-23 21:46:05 +03:00
Robert Schulze
71c4e3dbf4
A bit more memory-safe 2023-04-23 18:13:51 +00:00
Robert Schulze
38d5f12a32
Query Cache: Implement per-user quotas 2023-04-23 16:13:51 +00:00
Alexey Milovidov
c42e1ad131
Merge pull request #48922 from RunningXie/fix_reopen_stale_fd 
Implement a slight improvement for the rare case when ClickHouse is run inside LXC and LXCFS is used, because LXCFS has a bug
 2023-04-23 16:12:49 +03:00
kssenii
d8023806a9 Merge remote-tracking branch 'upstream/master' into better-local-object-storage 2023-04-23 12:39:34 +02:00
Alexey Milovidov
1be5a20546
Simplify the code 2023-04-23 09:01:21 +03:00
Alexey Milovidov
1087c7361d
Update AsynchronousMetrics.cpp 2023-04-23 08:51:17 +03:00
Alexey Milovidov
28e03f5781 Merge branch 'master' into fix-ibm 2023-04-23 06:01:41 +02:00
Sergei Trifonov
7fa5ccb53c
Merge branch 'master' into async-loader 2023-04-22 19:44:44 +02:00
serxa
056268f90b fix 2023-04-22 17:43:36 +00:00
serxa
64fb8da812 improve AsyncLoader interface 2023-04-22 17:29:53 +00:00
Alexey Milovidov
5fd42a2220 Add ProfileEvents for log messages 2023-04-22 14:55:25 +02:00
Alexey Milovidov
4c7f747fae
Update AsynchronousMetrics.cpp 2023-04-22 14:51:21 +03:00
Dmitry Novik
e23105b322 Merge remote-tracking branch 'origin/master' into revert-47476-revert_46622 2023-04-21 16:40:04 +00:00
Alexey Milovidov
a6fceaa497
Update AsynchronousMetrics.cpp 2023-04-21 13:51:13 +03:00
kssenii
e58c16318f Merge remote-tracking branch 'upstream/master' into better-local-object-storage 2023-04-21 12:49:40 +02:00
Alexey Milovidov
252c755323 Merge branch 'master' of github.com:ClickHouse/ClickHouse into fix-ibm 2023-04-21 12:39:13 +02:00
Alexey Milovidov
530b764953 Fix IBM 2023-04-21 12:38:45 +02:00
Alexey Milovidov
9d6c3d7a4c
Merge pull request #48920 from awfeequdng/bugfix/maskLowBits 
A non significant change (does not affect anything): add support for signed integers in the maskBits function
 2023-04-21 13:22:44 +03:00
Kruglov Pavel
9749448eb0
Merge pull request #47229 from Avogar/non-blocking-connect 
Improve async reading from socket
 2023-04-21 10:38:50 +02:00
pengxiangcai
41cf29179d typename spelling error: unsigedT -> UnsignedT 2023-04-21 08:51:58 +08:00
Alexey Milovidov
98ae9be734
Revert "Added tests for ClickHouse apps help and fixed help issues" 2023-04-21 01:54:34 +03:00
Arthur Passos
43841a6ee6 Merge branch 'master' into parsey_key_value_function 2023-04-20 10:05:34 -03:00
Antonio Andelic
31548ab17c Fix result 2023-04-20 12:30:24 +00:00
Antonio Andelic
09156ab2b7 Merge branch 'master' into keeper-check-not-exists 2023-04-20 12:07:16 +00:00
avogar
39ba4c27bc Fix deadlock, add profile event 2023-04-20 11:56:20 +00:00
Sergei Trifonov
4b51f12479
Update src/Common/AsyncLoader.h 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2023-04-19 13:48:38 +02:00
serxa
5390fccf50 Merge branch 'async-loader' of github.com:ClickHouse/ClickHouse into async-loader 2023-04-19 11:47:08 +00:00
serxa
2e31140ca4 review fixes 2023-04-19 11:46:56 +00:00
Kruglov Pavel
2ad161d2b7
Merge branch 'master' into non-blocking-connect 2023-04-19 13:39:40 +02:00
Sergei Trifonov
31e3a3bc0c
Update src/Common/AsyncLoader.h 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2023-04-19 13:35:58 +02:00
Antonio Andelic
58e9b56fcb Fix CheckNotExists 2023-04-19 09:06:20 +00:00
serxa
ff7444f077 typo 2023-04-19 06:27:17 +00:00
serxa
c672d117ea better job shutdown 2023-04-19 06:27:07 +00:00
serxa
8cc9b32af1 fix more comments 2023-04-19 06:26:59 +00:00
serxa
d4daea5f09 fix typos 2023-04-19 06:26:51 +00:00
serxa
baced2c66e fix data race in test 2023-04-19 06:26:35 +00:00
serxa
326c733be6 fix tidy build 2023-04-19 06:26:27 +00:00
serxa
54c5b1083b add test for dynamic prioritization 2023-04-19 06:26:14 +00:00
serxa
911c46aa32 fix double finish() during cancellation of a job with multiple paths 2023-04-19 06:26:06 +00:00
serxa
0d7e3d4100 fix comment 2023-04-19 06:25:54 +00:00
serxa
356b934835 split AsyncLoader into .h and .cpp files + more docs 2023-04-19 06:25:45 +00:00
Sergei Trifonov
1efacba196 Update src/Common/AsyncLoader.h 2023-04-19 06:25:37 +00:00
Sergei Trifonov
281b72c413 Update src/Common/AsyncLoader.h 2023-04-19 06:25:28 +00:00
serxa
1a3780ed8e fix 2023-04-19 06:25:11 +00:00
serxa
800efd5df8 add a good comment 2023-04-19 06:24:50 +00:00
serxa
69719c0819 allow to change max number of workers in runtime 2023-04-19 06:24:42 +00:00
serxa
c736d9fd4f fix test 2023-04-19 06:24:35 +00:00
serxa
264956fece add test for static priorities 2023-04-19 06:24:25 +00:00
serxa
b86c737450 make performance-inefficient-vector-operation,-warnings-as-errors happy 2023-04-19 06:24:15 +00:00
serxa
8c1f734f50 add random test with dependent tasks 2023-04-19 06:24:07 +00:00
serxa
d8bf775b9a fix worker shutdown 2023-04-19 06:24:01 +00:00
serxa
51992ae96d add ScheduleJobWithCanceledDependencies test 2023-04-19 06:23:47 +00:00
serxa
5837b09880 fix scheduling of jobs with dependencies on finished jobs 2023-04-19 06:23:39 +00:00
serxa
5ba1dc91e3 add LoadStatus::CANCELED 2023-04-19 06:23:32 +00:00
serxa
df3c5212ff test deps are completed + refactoring 2023-04-19 06:23:22 +00:00
serxa
9ac37ca3b8 add overload test 2023-04-19 06:23:14 +00:00
serxa
4bc52a1b06 add concurrency test 2023-04-19 06:23:07 +00:00
serxa
ba67f95525 allow cancels on RandomTasks test shutdown 2023-04-19 06:22:37 +00:00
serxa
3be0645413 fix remove of executing tasks + add test 2023-04-19 06:22:27 +00:00
serxa
a9c51c4aca add more tests for cancel 2023-04-19 06:22:15 +00:00
serxa
5e5a38cd64 add tests for job status/wait/cancel 2023-04-19 06:22:01 +00:00
serxa
3226eadd94 add simple randomized test 2023-04-19 06:21:31 +00:00
serxa
2c29c0b2f3 add dependency cycle detection 2023-04-19 06:21:19 +00:00
serxa
b4318b4ab6 fix style 2023-04-19 06:21:08 +00:00
serxa
5d11706929 make t work, add simple test 2023-04-19 06:20:37 +00:00
serxa
3fe36a3db6 wip AsyncLoader 2023-04-19 06:20:17 +00:00
caipengxiang
c9848e0822 bugfix: 
maskLowBits<UInt8>(3) => 00000111, but maskLowBits<Int8>(3) => 11111111, so I fixed it.
    after I fixed it, the result is:
    maskLowBits<UInt8>(3) => 00000111, maskLowBits<Int8>(3) => 00000111
 2023-04-19 12:22:17 +08:00
xieyichen
11b3469eac reopen stale fd 2023-04-19 11:13:16 +08:00
Ilya Yatsishin
ebb1b990f2
Merge pull request #45819 from qoega/clickhouse-help 2023-04-18 21:46:36 +02:00
Antonio Andelic
fb16623d48 Add CheckNotExists request to Keeper 2023-04-18 14:45:24 +00:00
kssenii
16b027ed0c Merge remote-tracking branch 'upstream/master' into better-local-object-storage 2023-04-18 16:25:08 +02:00
Kruglov Pavel
cd88024a33
Merge pull request #48556 from ClickHouse/fix-nested-map-ip-uuid 
Fix nested map for keys of IP and UUID types
 2023-04-18 13:03:50 +02:00
Alexander Gololobov
423193d619
Merge branch 'master' into fix_zookeeper_join_race 2023-04-17 20:50:51 +02:00
kssenii
a470deafbc Fix style check 2023-04-17 19:11:42 +02:00
Alexander Gololobov
0493b2cbce A wrapper to allow calling join() from multiple threads 2023-04-17 15:01:45 +02:00
Raúl Marín
39f8c43a60 Merge remote-tracking branch 'blessed/master' into arenita 2023-04-17 10:33:38 +02:00
Michael Kolupaev
e133633359 Parallel decoding with one row group per thread 2023-04-17 04:58:32 +00:00
Michael Kolupaev
2d4fe85513 Something 2023-04-17 04:58:32 +00:00
Dmitry Novik
a1f21a5fc4 Do not log peak for background memory tracker 2023-04-14 16:49:14 +02:00
Dmitry Novik
cf5d9a175a Revert "Merge pull request #48760 from ClickHouse/revert-46089-background-memory-tracker" 
This reverts commit a61ed33223, reversing
changes made to 5f01b8a2b5.
 2023-04-14 16:34:19 +02:00
Sergei Trifonov
7cb6303e4c
Merge branch 'master' into lower-threadpool-mutex-contention 2023-04-14 03:26:39 +02:00
Alexander Tokmakov
af1bf08663
Revert "Add MemoryTracker for the background tasks" 2023-04-13 21:05:02 +03:00
serxa
22b75ec81b fix typo, add coment, dont reset trace 2023-04-13 15:17:39 +00:00
Arthur Passos
88f1c969cc
Merge branch 'master' into parsey_key_value_function 2023-04-13 10:38:44 -03:00
Sergei Trifonov
c51f1359be
Merge branch 'master' into lower-threadpool-mutex-contention 2023-04-13 14:21:45 +02:00
serxa
32f78afd7e get rid of redundant code 2023-04-13 12:11:06 +00:00
serxa
ce2073e58f detach_thread() and notifications order does not matter 2023-04-13 11:45:19 +00:00
Raúl Marín
e239df4c25 Keep count to avoid iterating over lists 2023-04-13 13:41:44 +02:00
Dmitry Novik
467ecf45e1
Merge pull request #46089 from ClickHouse/background-memory-tracker 
Add MemoryTracker for the background tasks
 2023-04-13 13:29:27 +02:00
serxa
5e8339c2d9 Lock ThreadPool mutex once per job instead of twice 2023-04-13 11:27:11 +00:00
Raúl Marín
2b70e08f23 Don't count unreserved bytes in Arenas as read_bytes 2023-04-13 12:43:24 +02:00
Raúl Marín
da9a539cf7 Reduce the usage of Arena.h 2023-04-13 10:31:32 +02:00
Robert Schulze
22c3b10e41
Merge pull request #48628 from ClickHouse/rs/var_uint_max 
Better handling of values too large for VarInt encoding
 2023-04-13 10:17:55 +02:00
Robert Schulze
f41354ccd6
Merge pull request #48671 from ClickHouse/rs/gcc-removal 
Remove GCC remainders
 2023-04-13 10:15:35 +02:00
Robert Schulze
3f7ce60e03
Merge branch 'master' into rs/gcc-removal 2023-04-12 22:17:04 +02:00
Azat Khuzhin
37d430d68a Use ThreadPool in PipelineExecutor 
PipelineExecutor is one of the heavy user of the threads, by using pool
there number of used threads for query execution can be tracked in
system.metrics.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-12 20:00:51 +02:00
Robert Schulze
05606a8835
Clean up GCC warning pragmas 2023-04-11 18:21:08 +00:00
Dmitry Novik
52f3406bbb Fix CurrentMetrics::MergesMutationsMemoryTracking value 2023-04-11 16:10:59 +00:00
Dmitry Novik
06e6794fc0 Merge remote-tracking branch 'origin/master' into background-memory-tracker 2023-04-11 15:29:35 +00:00
Yatsishin Ilya
b5b65d2149 Merge remote-tracking branch 'origin' into clickhouse-help 2023-04-11 11:24:48 +00:00
Sema Checherinda
4dd86a406a
Merge pull request #48543 from azat/mv-uniq-thread-group 
Use one ThreadGroup while pushing to materialized views (and some refactoring for ThreadGroup)
 2023-04-11 11:47:46 +02:00
Robert Schulze
bbbb55cd50
Better handling of values too large for VarInt encoding 
PR #48154 introduced a sanity check in the form of a debug assertion
that the input values for VarInt encoding are not too big. Such values
should be exceptionally rare in practice but the AST fuzzer managed to
trigger the assertion regardless. The strategy to deal with such values
until now was to bypass the check by limiting the value to the maximum
allowed value (see #48412). Because a new AST Fuzzer failure appeared
(#48497) and there may be more failures in future, this PR changes the
sanity check from an assert to an exception.

Fixes: #48497
 2023-04-11 07:47:33 +00:00
Azat Khuzhin
79b83c4fd2 Remove superfluous includes of logger_userful.h from headers 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-10 17:59:30 +02:00
Arthur Passos
b062ab5b4f
Merge branch 'master' into parsey_key_value_function 2023-04-10 10:32:01 -03:00
Alexey Milovidov
6292a08c9c Add a test 2023-04-09 03:25:38 +02:00
Alexey Milovidov
09ea79aaf7 Add support for {server_uuid} macro 2023-04-09 03:04:26 +02:00
Alexey Milovidov
5495cb2935
Merge pull request #48521 from ClibMouse/feature/tiny-improvement 
Use std::string::starts_with instead of a roll your own variant
 2023-04-08 22:19:48 +03:00
Yakov Olkhovskiy
054078e06d fix nested map for keys of IP and UUID types 2023-04-08 05:34:35 +00:00
Vitaly Baranov
62e9c419de
Merge pull request #48519 from azat/build/fwd-decl-threadpool 
Use forward declaration of ThreadPool
 2023-04-08 00:44:01 +02:00
Antonio Andelic
55a999d6a3
Merge branch 'master' into fix-topk-ubsan 2023-04-07 20:05:36 +02:00
Azat Khuzhin
aacf2a0838 Move ThreadGroupSwitcher to ThreadStatus.h (out from MergeTree code) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-07 15:32:08 +02:00
Azat Khuzhin
5b2b20a0b0 Rename ThreadGroupStatus to ThreadGroup 
There are methods like getThreadGroup() and ThreadGroupSwitcher class,
so seems that this is logical.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-07 15:31:48 +02:00
alesapin
a0dd1c96bf
Merge pull request #48512 from ClickHouse/serxa-patch-3 
Fix bytesSize() of zk SetRequest
 2023-04-07 11:47:31 +02:00
Azat Khuzhin
011480924a Use forward declaration of ThreadPool 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-07 11:25:35 +02:00
Antonio Andelic
67e6335250 Fix nullptr to memcpy 2023-04-07 08:32:01 +00:00
robot-clickhouse-ci-2
2cb8252676
Merge pull request #48487 from azat/memory-tracker-fix-deadlock 
Fix deadlock due to debug tracking of memory allocations
 2023-04-06 23:50:44 +02:00
ltrk2
790e570af6 Use std::string::starts_with instead of a roll your own variant 2023-04-06 11:57:44 -07:00
Sergei Trifonov
b6975d36e9
Fix bytesSize() of zk SetRequest 2023-04-06 19:05:32 +02:00
Alexey Milovidov
4ab8b35b00
Merge pull request #48444 from ClickHouse/better-exception-messages-from-keeper-client 
Better exception messages from Keeper client
 2023-04-06 18:36:39 +03:00
Azat Khuzhin
e91c8bfea3 Fix deadlock due to debug tracking of memory allocations 
Since memory tracker should be blocked globally (by default it is blocked only
for VariableContext::User).

<details>

<summary>stacktrace</summary>

    * thread 11, name = 'Collector', stop reason = trace
        frame 1: 0x00007ffff7e37ea2 libc.so.6`pthread_mutex_lock + 274
        frame 6: 0x000000001ee10298 clickhouse`toStringCached(pointers=0x00007fffedfef068, offset=0, size=45) + 88 at StackTrace.cpp:425
        frame 7: 0x000000001ee1128e clickhouse`StackTrace::toString(this=0x00007fffedfef058) const + 46 at StackTrace.cpp:444
        frame 8: 0x000000001ede58ff clickhouse`(anonymous namespace)::debugLogBigAllocationWithoutCheck(size=25187648) + 479 at MemoryTracker.cpp:98
        frame 9: 0x000000001ede5481 clickhouse`MemoryTracker::allocImpl(this=0x0000000030729e68, size=25187648, throw_if_memory_exceeded=false, query_tracker=0x00007ffff3c22440) + 3937 at MemoryTracker.cpp:352
        frame 10: 0x000000001ede477f clickhouse`MemoryTracker::allocImpl(this=0x00007ffff3c1e530, size=25187648, throw_if_memory_exceeded=false, query_tracker=0x00007ffff3c22440) + 607 at MemoryTracker.cpp:198
        frame 11: 0x000000001ede477f clickhouse`MemoryTracker::allocImpl(this=0x00007ffff3c22440, size=25187648, throw_if_memory_exceeded=false, query_tracker=0x0000000000000000) + 607 at MemoryTracker.cpp:198
        frame 12: 0x000000001ede477f clickhouse`MemoryTracker::allocImpl(this=0x00007fffedff3bf0, size=25187648, throw_if_memory_exceeded=false, query_tracker=0x0000000000000000) + 607 at MemoryTracker.cpp:198
        frame 13: 0x000000001ed6c4c9 clickhouse`CurrentMemoryTracker::allocImpl(size=25165824, throw_if_memory_exceeded=false) + 265 at CurrentMemoryTracker.cpp:58
        frame 14: 0x000000001ed6c6a1 clickhouse`CurrentMemoryTracker::allocNoThrow(size=25165824) + 33 at CurrentMemoryTracker.cpp:91
        frame 15: 0x000000001ed63079 clickhouse`operator new(unsigned long) [inlined] void Memory::trackMemory<>(size=25165824) + 89 at memory.h:177
        frame 16: 0x000000001ed63034 clickhouse`operator new(size=25165824) + 20 at new_delete.cpp:74
        ...
        frame 27: 0x000000001ee15895 clickhouse`DB::(anonymous namespace)::collectSymbols(info=0x00007ffff7062000, (null)=<unavailable>, data_ptr=0x00007fffec2b7000) + 53 at SymbolIndex.cpp:483
        frame 28: 0x000000002e1a5d57 clickhouse`::dl_iterate_phdr(callback=(clickhouse`DB::(anonymous namespace)::collectSymbols(dl_phdr_info*, unsigned long, void*) at SymbolIndex.cpp:480), data=0x00007fffec2b7000)(dl_phdr_info *, size_t, void *), void *) + 183 at phdr_cache.cpp:76
        frame 29: 0x000000001ee15549 clickhouse`DB::SymbolIndex::update(this=0x00007fffec2b7000) + 41 at SymbolIndex.cpp:515
        frame 30: 0x000000001ee1be35 clickhouse`DB::SymbolIndex::SymbolIndex(this=<unavailable>) + 53 at SymbolIndex.h:23
        frame 31: 0x000000001ee17a6c clickhouse`DB::SymbolIndex::instanceImpl() + 108 at SymbolIndex.cpp:555
        frame 32: 0x000000001ee17b4a clickhouse`DB::SymbolIndex::instance() + 10 at SymbolIndex.cpp:561
        frame 33: 0x000000001ee1070f clickhouse`void toStringEveryLineImpl<toStringCached(std::__1::array<void*, 45ul> const&, unsigned long, unsigned long)::$_0>(fatal=false, stack_trace=0x00007fffedff1ef8, callback=0x00007fffedff1ee0)::$_0&&) + 207 at StackTrace.cpp:349
        frame 34: 0x000000001ee1040b clickhouse`toStringCached(pointers=0x00007fffedff2928, offset=0, size=43) + 459 at StackTrace.cpp:436
        frame 35: 0x000000001ee1128e clickhouse`StackTrace::toString(this=0x00007fffedff2918) const + 46 at StackTrace.cpp:444
        frame 36: 0x000000001ede58ff clickhouse`(anonymous namespace)::debugLogBigAllocationWithoutCheck(size=25165824) + 479 at MemoryTracker.cpp:98
        frame 37: 0x000000001ede5481 clickhouse`MemoryTracker::allocImpl(this=0x00007fffedff3bf0, size=25165824, throw_if_memory_exceeded=false, query_tracker=0x0000000000000000) + 3937 at MemoryTracker.cpp:352
        frame 38: 0x000000001ed6c4c9 clickhouse`CurrentMemoryTracker::allocImpl(size=25165824, throw_if_memory_exceeded=false) + 265 at CurrentMemoryTracker.cpp:58
        frame 39: 0x000000001ed6c6a1 clickhouse`CurrentMemoryTracker::allocNoThrow(size=25165824) + 33 at CurrentMemoryTracker.cpp:91
        frame 40: 0x000000001ed63079 clickhouse`operator new(unsigned long) [inlined] void Memory::trackMemory<>(size=25165824) + 89 at memory.h:177
        frame 41: 0x000000001ed63034 clickhouse`operator new(size=25165824) + 20 at new_delete.cpp:74
        ...
        frame 55: 0x0000000025badace clickhouse`DB::WriteBufferFromS3::nextImpl(this=0x00007ffff3c58f00) + 366 at WriteBufferFromS3.cpp:110

</details>

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-06 16:03:46 +02:00
Sema Checherinda
b8729be97a
Merge pull request #47104 from CheSema/thead-group-for-merge 
merge and mutation make thread group for setting memory trackers right
 2023-04-06 15:46:56 +02:00
Robert Schulze
ce2692ba79
Merge pull request #47246 from ClickHouse/rs/m 
Change the behavior of formatter %M in function formatDateTime() from minutes to month name
 2023-04-06 09:19:50 +02:00
Alexey Milovidov
785ea8213b Better exception messages from Keeper client 2023-04-06 02:29:37 +02:00
Alexander Tokmakov
a154fa73f4
Merge pull request #48401 from ClickHouse/fix_00002_log_messages 
Find big allocations without memory limits checks
 2023-04-05 23:52:39 +03:00
Robert Schulze
809a7fbb7a
Incorporate more review feedback 2023-04-05 19:43:30 +00:00
Sema Checherinda
554c928534
Merge branch 'master' into thead-group-for-merge 2023-04-05 21:09:38 +02:00
robot-clickhouse
3d2c279859
Merge pull request #48242 from azat/better-throttling 
Per-query/per-server throttling for remote IO/local IO/BACKUPs
 2023-04-05 19:32:37 +02:00
Alexander Tokmakov
0e0e89d596 fix 2023-04-05 15:10:09 +02:00
Robert Schulze
598b050628
AST Fuzzer: Fix assertion in TopK serialization 
Problem:
  https://s3.amazonaws.com/clickhouse-test-reports/0/fa5b2bd4a5b02336bca8837c473a7124f8ecedf2/fuzzer_astfuzzerasan/report.html

The new assertion in the Varint code was introduced with (*). It rejects
values whose serialization cannot be deserialized (and this behavior
cannot be changed due to historical reasons). Such values should be
exceptionally rare in practice but AST fuzzer managers to trigger them.

The fix is similar to (**): Bypass the check by limiting the value to
the maximum allowed value.

(if AST fuzzer triggers finds more violations of the assertion, we might
consider throwing an exception instead)

(*) https://github.com/ClickHouse/ClickHouse/pull/48154
(**) https://github.com/ClickHouse/ClickHouse/pull/48154/files#diff-653c0a18dfdaa86262c78dc6b25550add0487f165b4ad053e86f530388f6203a
 2023-04-05 08:07:37 +00:00
Azat Khuzhin
a25dd1d348 Add ability to throttle local IO on per-query/server basis 
Server settings:
- max_local_read_bandwidth_for_server
- max_local_write_bandwidth_for_server

Query settings:
- max_local_read_bandwidth
- max_local_write_bandwidth

This is the preparation for adding ability to throttle BACKUPs

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-05 09:39:15 +02:00
Azat Khuzhin
b3406beeb7 Add per-query network throttling 
Controlled with:
- max_remote_read_network_bandwidth
- max_remote_write_network_bandwidth

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-05 09:39:15 +02:00
Arthur Passos
34be1893fd fix unit tests 2023-04-04 20:31:12 -03:00
Arthur Passos
474377c16f remove debug find functions and replace SearchSymbols assertion with an exception, check with upstream preferred way 2023-04-04 16:14:56 -03:00
Alexander Tokmakov
f5574d0d74 find big allocations without limits checks 2023-04-04 20:10:22 +02:00
Nikita Mikhaylov
fa5b2bd4a5
Added Keeper retries for backups operations (#47224) 2023-04-04 18:12:08 +02:00
Arthur Passos
fa1fdf7573 style 2023-04-03 15:28:17 -03:00
Arthur Passos
c80ba3d52e unit tests passing 2023-04-03 13:48:58 -03:00
Antonio Andelic
f5e97fbb66 Merge branch 'master' into alternative-keeper-configs 2023-04-03 15:06:52 +00:00
Azat Khuzhin
fbc99d7788 Use StrongTypedef for StatusInfo 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-01 16:01:33 +02:00
Azat Khuzhin
c64f9e6f07 Use StrongTypedef for ProfileEvents 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-01 16:01:33 +02:00
Azat Khuzhin
8d0e516310 Use StrongTypedef for CurrentMetrics to avoid possible incorrect usage 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-01 16:01:33 +02:00
Azat Khuzhin
f69441b633 Fix ThreadPool usage in gtest_thread_pool_limit 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-01 16:00:03 +02:00
Azat Khuzhin
ba6ecd2d4e Fix ThreadPool for DistributedSink 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-01 16:00:03 +02:00
Alexander Tokmakov
29e85bd9a4
Merge branch 'master' into fix-threadpools 2023-03-31 01:03:06 +03:00
Sergei Trifonov
01ed4ffca4
Merge pull request #47870 from ClickHouse/segmentator-fix 
Fix CPU usage counters update logic
 2023-03-30 22:55:58 +02:00
Azat Khuzhin
edeeac84cf Fix incorrect ThreadPool usage after ThreadPool introspection 
```
$ gg 'ThreadPool[^()]([A-Za-z_]\+,' src/
src/Interpreters/Context.cpp:        shared->load_marks_threadpool = std::make_unique<ThreadPool>(pool_size, pool_size, queue_size);
src/Interpreters/Context.cpp:        shared->prefetch_threadpool = std::make_unique<ThreadPool>(pool_size, pool_size, queue_size);
src/Interpreters/Context.cpp:        shared->threadpool_writer = std::make_unique<ThreadPool>(pool_size, pool_size, queue_size);
```

Fixes: #47880
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-03-30 20:08:38 +02:00
Alexey Milovidov
e982fb9f1c
Merge pull request #47880 from azat/threadpool-introspection 
ThreadPool metrics introspection
 2023-03-30 01:27:31 +03:00
Alexey Milovidov
13761fe891
Merge pull request #48133 from ClickHouse/allow-to-skip-errors-of-enums 
Allow skipping errors related to unknown enum values in row input formats
 2023-03-29 23:20:05 +03:00
Frank Chen
87d235e842
Some improvements about names of span logs (#47667) 2023-03-29 16:28:36 +02:00
Azat Khuzhin
f38a7aeabe ThreadPool metrics introspection 
There are lots of thread pools and simple local-vs-global is not enough
already, it is good to know which one in particular uses threads.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-03-29 10:46:59 +02:00
Sergei Trifonov
1ff6519c25
Merge branch 'master' into segmentator-fix 2023-03-29 02:29:43 +02:00
serxa
81af0b6deb avoid counters updates if not initialized 2023-03-29 00:29:21 +00:00
Alexey Milovidov
cfc4e40811 Allow skipping errors related to unknown enum values in row input formats 2023-03-29 00:31:53 +02:00
Alexander Tokmakov
614f8b313e
Merge pull request #47856 from Algunenano/zk_inject_timeout 
Allow injecting timeout errors on Keeper
 2023-03-28 23:40:51 +03:00
Yatsishin Ilya
26aacd3d2d Merge remote-tracking branch 'origin/master' into clickhouse-help 2023-03-28 16:57:58 +00:00
avogar
e63231700f Fix tests 2023-03-28 16:00:00 +00:00
Sema Checherinda
b774310dc9 keep query id for logging when optimize query 2023-03-28 16:44:47 +02:00
Sema Checherinda
10f1030a14
Merge pull request #47147 from aletik256/fix_JSON_searchField 
fix_JSON_searchField
 2023-03-28 14:45:09 +02:00