thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-16 03:12:43 +00:00
Code Issues Packages Projects Releases Wiki Activity
115,637 Commits 9,326 Branches 1,783 Tags 2.1 GiB
092b06f32e
Commit Graph

5763 Commits

Author SHA1 Message Date
serxa
056268f90b fix 2023-04-22 17:43:36 +00:00
serxa
64fb8da812 improve AsyncLoader interface 2023-04-22 17:29:53 +00:00
Alexey Milovidov
5fd42a2220 Add ProfileEvents for log messages 2023-04-22 14:55:25 +02:00
Alexey Milovidov
4c7f747fae
Update AsynchronousMetrics.cpp 2023-04-22 14:51:21 +03:00
Dmitry Novik
e23105b322 Merge remote-tracking branch 'origin/master' into revert-47476-revert_46622 2023-04-21 16:40:04 +00:00
Alexey Milovidov
a6fceaa497
Update AsynchronousMetrics.cpp 2023-04-21 13:51:13 +03:00
kssenii
e58c16318f Merge remote-tracking branch 'upstream/master' into better-local-object-storage 2023-04-21 12:49:40 +02:00
Alexey Milovidov
252c755323 Merge branch 'master' of github.com:ClickHouse/ClickHouse into fix-ibm 2023-04-21 12:39:13 +02:00
Alexey Milovidov
530b764953 Fix IBM 2023-04-21 12:38:45 +02:00
Alexey Milovidov
9d6c3d7a4c
Merge pull request #48920 from awfeequdng/bugfix/maskLowBits 
A non significant change (does not affect anything): add support for signed integers in the maskBits function
 2023-04-21 13:22:44 +03:00
Kruglov Pavel
9749448eb0
Merge pull request #47229 from Avogar/non-blocking-connect 
Improve async reading from socket
 2023-04-21 10:38:50 +02:00
pengxiangcai
41cf29179d typename spelling error: unsigedT -> UnsignedT 2023-04-21 08:51:58 +08:00
Alexey Milovidov
98ae9be734
Revert "Added tests for ClickHouse apps help and fixed help issues" 2023-04-21 01:54:34 +03:00
Arthur Passos
43841a6ee6 Merge branch 'master' into parsey_key_value_function 2023-04-20 10:05:34 -03:00
Antonio Andelic
31548ab17c Fix result 2023-04-20 12:30:24 +00:00
Antonio Andelic
09156ab2b7 Merge branch 'master' into keeper-check-not-exists 2023-04-20 12:07:16 +00:00
avogar
39ba4c27bc Fix deadlock, add profile event 2023-04-20 11:56:20 +00:00
Sergei Trifonov
4b51f12479
Update src/Common/AsyncLoader.h 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2023-04-19 13:48:38 +02:00
serxa
5390fccf50 Merge branch 'async-loader' of github.com:ClickHouse/ClickHouse into async-loader 2023-04-19 11:47:08 +00:00
serxa
2e31140ca4 review fixes 2023-04-19 11:46:56 +00:00
Kruglov Pavel
2ad161d2b7
Merge branch 'master' into non-blocking-connect 2023-04-19 13:39:40 +02:00
Sergei Trifonov
31e3a3bc0c
Update src/Common/AsyncLoader.h 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2023-04-19 13:35:58 +02:00
Antonio Andelic
58e9b56fcb Fix CheckNotExists 2023-04-19 09:06:20 +00:00
serxa
ff7444f077 typo 2023-04-19 06:27:17 +00:00
serxa
c672d117ea better job shutdown 2023-04-19 06:27:07 +00:00
serxa
8cc9b32af1 fix more comments 2023-04-19 06:26:59 +00:00
serxa
d4daea5f09 fix typos 2023-04-19 06:26:51 +00:00
serxa
baced2c66e fix data race in test 2023-04-19 06:26:35 +00:00
serxa
326c733be6 fix tidy build 2023-04-19 06:26:27 +00:00
serxa
54c5b1083b add test for dynamic prioritization 2023-04-19 06:26:14 +00:00
serxa
911c46aa32 fix double finish() during cancellation of a job with multiple paths 2023-04-19 06:26:06 +00:00
serxa
0d7e3d4100 fix comment 2023-04-19 06:25:54 +00:00
serxa
356b934835 split AsyncLoader into .h and .cpp files + more docs 2023-04-19 06:25:45 +00:00
Sergei Trifonov
1efacba196 Update src/Common/AsyncLoader.h 2023-04-19 06:25:37 +00:00
Sergei Trifonov
281b72c413 Update src/Common/AsyncLoader.h 2023-04-19 06:25:28 +00:00
serxa
1a3780ed8e fix 2023-04-19 06:25:11 +00:00
serxa
800efd5df8 add a good comment 2023-04-19 06:24:50 +00:00
serxa
69719c0819 allow to change max number of workers in runtime 2023-04-19 06:24:42 +00:00
serxa
c736d9fd4f fix test 2023-04-19 06:24:35 +00:00
serxa
264956fece add test for static priorities 2023-04-19 06:24:25 +00:00
serxa
b86c737450 make performance-inefficient-vector-operation,-warnings-as-errors happy 2023-04-19 06:24:15 +00:00
serxa
8c1f734f50 add random test with dependent tasks 2023-04-19 06:24:07 +00:00
serxa
d8bf775b9a fix worker shutdown 2023-04-19 06:24:01 +00:00
serxa
51992ae96d add ScheduleJobWithCanceledDependencies test 2023-04-19 06:23:47 +00:00
serxa
5837b09880 fix scheduling of jobs with dependencies on finished jobs 2023-04-19 06:23:39 +00:00
serxa
5ba1dc91e3 add LoadStatus::CANCELED 2023-04-19 06:23:32 +00:00
serxa
df3c5212ff test deps are completed + refactoring 2023-04-19 06:23:22 +00:00
serxa
9ac37ca3b8 add overload test 2023-04-19 06:23:14 +00:00
serxa
4bc52a1b06 add concurrency test 2023-04-19 06:23:07 +00:00
serxa
ba67f95525 allow cancels on RandomTasks test shutdown 2023-04-19 06:22:37 +00:00
serxa
3be0645413 fix remove of executing tasks + add test 2023-04-19 06:22:27 +00:00
serxa
a9c51c4aca add more tests for cancel 2023-04-19 06:22:15 +00:00
serxa
5e5a38cd64 add tests for job status/wait/cancel 2023-04-19 06:22:01 +00:00
serxa
3226eadd94 add simple randomized test 2023-04-19 06:21:31 +00:00
serxa
2c29c0b2f3 add dependency cycle detection 2023-04-19 06:21:19 +00:00
serxa
b4318b4ab6 fix style 2023-04-19 06:21:08 +00:00
serxa
5d11706929 make t work, add simple test 2023-04-19 06:20:37 +00:00
serxa
3fe36a3db6 wip AsyncLoader 2023-04-19 06:20:17 +00:00
caipengxiang
c9848e0822 bugfix: 
maskLowBits<UInt8>(3) => 00000111, but maskLowBits<Int8>(3) => 11111111, so I fixed it.
    after I fixed it, the result is:
    maskLowBits<UInt8>(3) => 00000111, maskLowBits<Int8>(3) => 00000111
 2023-04-19 12:22:17 +08:00
xieyichen
11b3469eac reopen stale fd 2023-04-19 11:13:16 +08:00
Ilya Yatsishin
ebb1b990f2
Merge pull request #45819 from qoega/clickhouse-help 2023-04-18 21:46:36 +02:00
Antonio Andelic
fb16623d48 Add CheckNotExists request to Keeper 2023-04-18 14:45:24 +00:00
kssenii
16b027ed0c Merge remote-tracking branch 'upstream/master' into better-local-object-storage 2023-04-18 16:25:08 +02:00
Kruglov Pavel
cd88024a33
Merge pull request #48556 from ClickHouse/fix-nested-map-ip-uuid 
Fix nested map for keys of IP and UUID types
 2023-04-18 13:03:50 +02:00
Alexander Gololobov
423193d619
Merge branch 'master' into fix_zookeeper_join_race 2023-04-17 20:50:51 +02:00
kssenii
a470deafbc Fix style check 2023-04-17 19:11:42 +02:00
Alexander Gololobov
0493b2cbce A wrapper to allow calling join() from multiple threads 2023-04-17 15:01:45 +02:00
Raúl Marín
39f8c43a60 Merge remote-tracking branch 'blessed/master' into arenita 2023-04-17 10:33:38 +02:00
Michael Kolupaev
e133633359 Parallel decoding with one row group per thread 2023-04-17 04:58:32 +00:00
Michael Kolupaev
2d4fe85513 Something 2023-04-17 04:58:32 +00:00
Dmitry Novik
a1f21a5fc4 Do not log peak for background memory tracker 2023-04-14 16:49:14 +02:00
Dmitry Novik
cf5d9a175a Revert "Merge pull request #48760 from ClickHouse/revert-46089-background-memory-tracker" 
This reverts commit a61ed33223, reversing
changes made to 5f01b8a2b5.
 2023-04-14 16:34:19 +02:00
Sergei Trifonov
7cb6303e4c
Merge branch 'master' into lower-threadpool-mutex-contention 2023-04-14 03:26:39 +02:00
Alexander Tokmakov
af1bf08663
Revert "Add MemoryTracker for the background tasks" 2023-04-13 21:05:02 +03:00
serxa
22b75ec81b fix typo, add coment, dont reset trace 2023-04-13 15:17:39 +00:00
Arthur Passos
88f1c969cc
Merge branch 'master' into parsey_key_value_function 2023-04-13 10:38:44 -03:00
Sergei Trifonov
c51f1359be
Merge branch 'master' into lower-threadpool-mutex-contention 2023-04-13 14:21:45 +02:00
serxa
32f78afd7e get rid of redundant code 2023-04-13 12:11:06 +00:00
serxa
ce2073e58f detach_thread() and notifications order does not matter 2023-04-13 11:45:19 +00:00
Raúl Marín
e239df4c25 Keep count to avoid iterating over lists 2023-04-13 13:41:44 +02:00
Dmitry Novik
467ecf45e1
Merge pull request #46089 from ClickHouse/background-memory-tracker 
Add MemoryTracker for the background tasks
 2023-04-13 13:29:27 +02:00
serxa
5e8339c2d9 Lock ThreadPool mutex once per job instead of twice 2023-04-13 11:27:11 +00:00
Raúl Marín
2b70e08f23 Don't count unreserved bytes in Arenas as read_bytes 2023-04-13 12:43:24 +02:00
Raúl Marín
da9a539cf7 Reduce the usage of Arena.h 2023-04-13 10:31:32 +02:00
Robert Schulze
22c3b10e41
Merge pull request #48628 from ClickHouse/rs/var_uint_max 
Better handling of values too large for VarInt encoding
 2023-04-13 10:17:55 +02:00
Robert Schulze
f41354ccd6
Merge pull request #48671 from ClickHouse/rs/gcc-removal 
Remove GCC remainders
 2023-04-13 10:15:35 +02:00
Robert Schulze
3f7ce60e03
Merge branch 'master' into rs/gcc-removal 2023-04-12 22:17:04 +02:00
Azat Khuzhin
37d430d68a Use ThreadPool in PipelineExecutor 
PipelineExecutor is one of the heavy user of the threads, by using pool
there number of used threads for query execution can be tracked in
system.metrics.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-12 20:00:51 +02:00
Robert Schulze
05606a8835
Clean up GCC warning pragmas 2023-04-11 18:21:08 +00:00
Dmitry Novik
52f3406bbb Fix CurrentMetrics::MergesMutationsMemoryTracking value 2023-04-11 16:10:59 +00:00
Dmitry Novik
06e6794fc0 Merge remote-tracking branch 'origin/master' into background-memory-tracker 2023-04-11 15:29:35 +00:00
Yatsishin Ilya
b5b65d2149 Merge remote-tracking branch 'origin' into clickhouse-help 2023-04-11 11:24:48 +00:00
Sema Checherinda
4dd86a406a
Merge pull request #48543 from azat/mv-uniq-thread-group 
Use one ThreadGroup while pushing to materialized views (and some refactoring for ThreadGroup)
 2023-04-11 11:47:46 +02:00
Robert Schulze
bbbb55cd50
Better handling of values too large for VarInt encoding 
PR #48154 introduced a sanity check in the form of a debug assertion
that the input values for VarInt encoding are not too big. Such values
should be exceptionally rare in practice but the AST fuzzer managed to
trigger the assertion regardless. The strategy to deal with such values
until now was to bypass the check by limiting the value to the maximum
allowed value (see #48412). Because a new AST Fuzzer failure appeared
(#48497) and there may be more failures in future, this PR changes the
sanity check from an assert to an exception.

Fixes: #48497
 2023-04-11 07:47:33 +00:00
Azat Khuzhin
79b83c4fd2 Remove superfluous includes of logger_userful.h from headers 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-10 17:59:30 +02:00
Arthur Passos
b062ab5b4f
Merge branch 'master' into parsey_key_value_function 2023-04-10 10:32:01 -03:00
Alexey Milovidov
6292a08c9c Add a test 2023-04-09 03:25:38 +02:00
Alexey Milovidov
09ea79aaf7 Add support for {server_uuid} macro 2023-04-09 03:04:26 +02:00
Alexey Milovidov
5495cb2935
Merge pull request #48521 from ClibMouse/feature/tiny-improvement 
Use std::string::starts_with instead of a roll your own variant
 2023-04-08 22:19:48 +03:00
Yakov Olkhovskiy
054078e06d fix nested map for keys of IP and UUID types 2023-04-08 05:34:35 +00:00
Vitaly Baranov
62e9c419de
Merge pull request #48519 from azat/build/fwd-decl-threadpool 
Use forward declaration of ThreadPool
 2023-04-08 00:44:01 +02:00
Antonio Andelic
55a999d6a3
Merge branch 'master' into fix-topk-ubsan 2023-04-07 20:05:36 +02:00
Azat Khuzhin
aacf2a0838 Move ThreadGroupSwitcher to ThreadStatus.h (out from MergeTree code) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-07 15:32:08 +02:00
Azat Khuzhin
5b2b20a0b0 Rename ThreadGroupStatus to ThreadGroup 
There are methods like getThreadGroup() and ThreadGroupSwitcher class,
so seems that this is logical.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-07 15:31:48 +02:00
alesapin
a0dd1c96bf
Merge pull request #48512 from ClickHouse/serxa-patch-3 
Fix bytesSize() of zk SetRequest
 2023-04-07 11:47:31 +02:00
Azat Khuzhin
011480924a Use forward declaration of ThreadPool 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-07 11:25:35 +02:00
Antonio Andelic
67e6335250 Fix nullptr to memcpy 2023-04-07 08:32:01 +00:00
robot-clickhouse-ci-2
2cb8252676
Merge pull request #48487 from azat/memory-tracker-fix-deadlock 
Fix deadlock due to debug tracking of memory allocations
 2023-04-06 23:50:44 +02:00
ltrk2
790e570af6 Use std::string::starts_with instead of a roll your own variant 2023-04-06 11:57:44 -07:00
Sergei Trifonov
b6975d36e9
Fix bytesSize() of zk SetRequest 2023-04-06 19:05:32 +02:00
Alexey Milovidov
4ab8b35b00
Merge pull request #48444 from ClickHouse/better-exception-messages-from-keeper-client 
Better exception messages from Keeper client
 2023-04-06 18:36:39 +03:00
Azat Khuzhin
e91c8bfea3 Fix deadlock due to debug tracking of memory allocations 
Since memory tracker should be blocked globally (by default it is blocked only
for VariableContext::User).

<details>

<summary>stacktrace</summary>

    * thread 11, name = 'Collector', stop reason = trace
        frame 1: 0x00007ffff7e37ea2 libc.so.6`pthread_mutex_lock + 274
        frame 6: 0x000000001ee10298 clickhouse`toStringCached(pointers=0x00007fffedfef068, offset=0, size=45) + 88 at StackTrace.cpp:425
        frame 7: 0x000000001ee1128e clickhouse`StackTrace::toString(this=0x00007fffedfef058) const + 46 at StackTrace.cpp:444
        frame 8: 0x000000001ede58ff clickhouse`(anonymous namespace)::debugLogBigAllocationWithoutCheck(size=25187648) + 479 at MemoryTracker.cpp:98
        frame 9: 0x000000001ede5481 clickhouse`MemoryTracker::allocImpl(this=0x0000000030729e68, size=25187648, throw_if_memory_exceeded=false, query_tracker=0x00007ffff3c22440) + 3937 at MemoryTracker.cpp:352
        frame 10: 0x000000001ede477f clickhouse`MemoryTracker::allocImpl(this=0x00007ffff3c1e530, size=25187648, throw_if_memory_exceeded=false, query_tracker=0x00007ffff3c22440) + 607 at MemoryTracker.cpp:198
        frame 11: 0x000000001ede477f clickhouse`MemoryTracker::allocImpl(this=0x00007ffff3c22440, size=25187648, throw_if_memory_exceeded=false, query_tracker=0x0000000000000000) + 607 at MemoryTracker.cpp:198
        frame 12: 0x000000001ede477f clickhouse`MemoryTracker::allocImpl(this=0x00007fffedff3bf0, size=25187648, throw_if_memory_exceeded=false, query_tracker=0x0000000000000000) + 607 at MemoryTracker.cpp:198
        frame 13: 0x000000001ed6c4c9 clickhouse`CurrentMemoryTracker::allocImpl(size=25165824, throw_if_memory_exceeded=false) + 265 at CurrentMemoryTracker.cpp:58
        frame 14: 0x000000001ed6c6a1 clickhouse`CurrentMemoryTracker::allocNoThrow(size=25165824) + 33 at CurrentMemoryTracker.cpp:91
        frame 15: 0x000000001ed63079 clickhouse`operator new(unsigned long) [inlined] void Memory::trackMemory<>(size=25165824) + 89 at memory.h:177
        frame 16: 0x000000001ed63034 clickhouse`operator new(size=25165824) + 20 at new_delete.cpp:74
        ...
        frame 27: 0x000000001ee15895 clickhouse`DB::(anonymous namespace)::collectSymbols(info=0x00007ffff7062000, (null)=<unavailable>, data_ptr=0x00007fffec2b7000) + 53 at SymbolIndex.cpp:483
        frame 28: 0x000000002e1a5d57 clickhouse`::dl_iterate_phdr(callback=(clickhouse`DB::(anonymous namespace)::collectSymbols(dl_phdr_info*, unsigned long, void*) at SymbolIndex.cpp:480), data=0x00007fffec2b7000)(dl_phdr_info *, size_t, void *), void *) + 183 at phdr_cache.cpp:76
        frame 29: 0x000000001ee15549 clickhouse`DB::SymbolIndex::update(this=0x00007fffec2b7000) + 41 at SymbolIndex.cpp:515
        frame 30: 0x000000001ee1be35 clickhouse`DB::SymbolIndex::SymbolIndex(this=<unavailable>) + 53 at SymbolIndex.h:23
        frame 31: 0x000000001ee17a6c clickhouse`DB::SymbolIndex::instanceImpl() + 108 at SymbolIndex.cpp:555
        frame 32: 0x000000001ee17b4a clickhouse`DB::SymbolIndex::instance() + 10 at SymbolIndex.cpp:561
        frame 33: 0x000000001ee1070f clickhouse`void toStringEveryLineImpl<toStringCached(std::__1::array<void*, 45ul> const&, unsigned long, unsigned long)::$_0>(fatal=false, stack_trace=0x00007fffedff1ef8, callback=0x00007fffedff1ee0)::$_0&&) + 207 at StackTrace.cpp:349
        frame 34: 0x000000001ee1040b clickhouse`toStringCached(pointers=0x00007fffedff2928, offset=0, size=43) + 459 at StackTrace.cpp:436
        frame 35: 0x000000001ee1128e clickhouse`StackTrace::toString(this=0x00007fffedff2918) const + 46 at StackTrace.cpp:444
        frame 36: 0x000000001ede58ff clickhouse`(anonymous namespace)::debugLogBigAllocationWithoutCheck(size=25165824) + 479 at MemoryTracker.cpp:98
        frame 37: 0x000000001ede5481 clickhouse`MemoryTracker::allocImpl(this=0x00007fffedff3bf0, size=25165824, throw_if_memory_exceeded=false, query_tracker=0x0000000000000000) + 3937 at MemoryTracker.cpp:352
        frame 38: 0x000000001ed6c4c9 clickhouse`CurrentMemoryTracker::allocImpl(size=25165824, throw_if_memory_exceeded=false) + 265 at CurrentMemoryTracker.cpp:58
        frame 39: 0x000000001ed6c6a1 clickhouse`CurrentMemoryTracker::allocNoThrow(size=25165824) + 33 at CurrentMemoryTracker.cpp:91
        frame 40: 0x000000001ed63079 clickhouse`operator new(unsigned long) [inlined] void Memory::trackMemory<>(size=25165824) + 89 at memory.h:177
        frame 41: 0x000000001ed63034 clickhouse`operator new(size=25165824) + 20 at new_delete.cpp:74
        ...
        frame 55: 0x0000000025badace clickhouse`DB::WriteBufferFromS3::nextImpl(this=0x00007ffff3c58f00) + 366 at WriteBufferFromS3.cpp:110

</details>

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-06 16:03:46 +02:00
Sema Checherinda
b8729be97a
Merge pull request #47104 from CheSema/thead-group-for-merge 
merge and mutation make thread group for setting memory trackers right
 2023-04-06 15:46:56 +02:00
Robert Schulze
ce2692ba79
Merge pull request #47246 from ClickHouse/rs/m 
Change the behavior of formatter %M in function formatDateTime() from minutes to month name
 2023-04-06 09:19:50 +02:00
Alexey Milovidov
785ea8213b Better exception messages from Keeper client 2023-04-06 02:29:37 +02:00
Alexander Tokmakov
a154fa73f4
Merge pull request #48401 from ClickHouse/fix_00002_log_messages 
Find big allocations without memory limits checks
 2023-04-05 23:52:39 +03:00
Robert Schulze
809a7fbb7a
Incorporate more review feedback 2023-04-05 19:43:30 +00:00
Sema Checherinda
554c928534
Merge branch 'master' into thead-group-for-merge 2023-04-05 21:09:38 +02:00
robot-clickhouse
3d2c279859
Merge pull request #48242 from azat/better-throttling 
Per-query/per-server throttling for remote IO/local IO/BACKUPs
 2023-04-05 19:32:37 +02:00
Alexander Tokmakov
0e0e89d596 fix 2023-04-05 15:10:09 +02:00
Robert Schulze
598b050628
AST Fuzzer: Fix assertion in TopK serialization 
Problem:
  https://s3.amazonaws.com/clickhouse-test-reports/0/fa5b2bd4a5b02336bca8837c473a7124f8ecedf2/fuzzer_astfuzzerasan/report.html

The new assertion in the Varint code was introduced with (*). It rejects
values whose serialization cannot be deserialized (and this behavior
cannot be changed due to historical reasons). Such values should be
exceptionally rare in practice but AST fuzzer managers to trigger them.

The fix is similar to (**): Bypass the check by limiting the value to
the maximum allowed value.

(if AST fuzzer triggers finds more violations of the assertion, we might
consider throwing an exception instead)

(*) https://github.com/ClickHouse/ClickHouse/pull/48154
(**) https://github.com/ClickHouse/ClickHouse/pull/48154/files#diff-653c0a18dfdaa86262c78dc6b25550add0487f165b4ad053e86f530388f6203a
 2023-04-05 08:07:37 +00:00
Azat Khuzhin
a25dd1d348 Add ability to throttle local IO on per-query/server basis 
Server settings:
- max_local_read_bandwidth_for_server
- max_local_write_bandwidth_for_server

Query settings:
- max_local_read_bandwidth
- max_local_write_bandwidth

This is the preparation for adding ability to throttle BACKUPs

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-05 09:39:15 +02:00
Azat Khuzhin
b3406beeb7 Add per-query network throttling 
Controlled with:
- max_remote_read_network_bandwidth
- max_remote_write_network_bandwidth

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-05 09:39:15 +02:00
Arthur Passos
34be1893fd fix unit tests 2023-04-04 20:31:12 -03:00
Arthur Passos
474377c16f remove debug find functions and replace SearchSymbols assertion with an exception, check with upstream preferred way 2023-04-04 16:14:56 -03:00
Alexander Tokmakov
f5574d0d74 find big allocations without limits checks 2023-04-04 20:10:22 +02:00
Nikita Mikhaylov
fa5b2bd4a5
Added Keeper retries for backups operations (#47224) 2023-04-04 18:12:08 +02:00
Arthur Passos
fa1fdf7573 style 2023-04-03 15:28:17 -03:00
Arthur Passos
c80ba3d52e unit tests passing 2023-04-03 13:48:58 -03:00
Antonio Andelic
f5e97fbb66 Merge branch 'master' into alternative-keeper-configs 2023-04-03 15:06:52 +00:00
Azat Khuzhin
fbc99d7788 Use StrongTypedef for StatusInfo 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-01 16:01:33 +02:00
Azat Khuzhin
c64f9e6f07 Use StrongTypedef for ProfileEvents 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-01 16:01:33 +02:00
Azat Khuzhin
8d0e516310 Use StrongTypedef for CurrentMetrics to avoid possible incorrect usage 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-01 16:01:33 +02:00
Azat Khuzhin
f69441b633 Fix ThreadPool usage in gtest_thread_pool_limit 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-01 16:00:03 +02:00
Azat Khuzhin
ba6ecd2d4e Fix ThreadPool for DistributedSink 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-01 16:00:03 +02:00
Alexander Tokmakov
29e85bd9a4
Merge branch 'master' into fix-threadpools 2023-03-31 01:03:06 +03:00
Sergei Trifonov
01ed4ffca4
Merge pull request #47870 from ClickHouse/segmentator-fix 
Fix CPU usage counters update logic
 2023-03-30 22:55:58 +02:00
Azat Khuzhin
edeeac84cf Fix incorrect ThreadPool usage after ThreadPool introspection 
```
$ gg 'ThreadPool[^()]([A-Za-z_]\+,' src/
src/Interpreters/Context.cpp:        shared->load_marks_threadpool = std::make_unique<ThreadPool>(pool_size, pool_size, queue_size);
src/Interpreters/Context.cpp:        shared->prefetch_threadpool = std::make_unique<ThreadPool>(pool_size, pool_size, queue_size);
src/Interpreters/Context.cpp:        shared->threadpool_writer = std::make_unique<ThreadPool>(pool_size, pool_size, queue_size);
```

Fixes: #47880
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-03-30 20:08:38 +02:00
Alexey Milovidov
e982fb9f1c
Merge pull request #47880 from azat/threadpool-introspection 
ThreadPool metrics introspection
 2023-03-30 01:27:31 +03:00
Alexey Milovidov
13761fe891
Merge pull request #48133 from ClickHouse/allow-to-skip-errors-of-enums 
Allow skipping errors related to unknown enum values in row input formats
 2023-03-29 23:20:05 +03:00
Frank Chen
87d235e842
Some improvements about names of span logs (#47667) 2023-03-29 16:28:36 +02:00
Azat Khuzhin
f38a7aeabe ThreadPool metrics introspection 
There are lots of thread pools and simple local-vs-global is not enough
already, it is good to know which one in particular uses threads.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-03-29 10:46:59 +02:00
Sergei Trifonov
1ff6519c25
Merge branch 'master' into segmentator-fix 2023-03-29 02:29:43 +02:00
serxa
81af0b6deb avoid counters updates if not initialized 2023-03-29 00:29:21 +00:00
Alexey Milovidov
cfc4e40811 Allow skipping errors related to unknown enum values in row input formats 2023-03-29 00:31:53 +02:00
Alexander Tokmakov
614f8b313e
Merge pull request #47856 from Algunenano/zk_inject_timeout 
Allow injecting timeout errors on Keeper
 2023-03-28 23:40:51 +03:00
Yatsishin Ilya
26aacd3d2d Merge remote-tracking branch 'origin/master' into clickhouse-help 2023-03-28 16:57:58 +00:00
avogar
e63231700f Fix tests 2023-03-28 16:00:00 +00:00
Sema Checherinda
b774310dc9 keep query id for logging when optimize query 2023-03-28 16:44:47 +02:00
Sema Checherinda
10f1030a14
Merge pull request #47147 from aletik256/fix_JSON_searchField 
fix_JSON_searchField
 2023-03-28 14:45:09 +02:00
Antonio Andelic
a0b6cd63bb fix build 2023-03-28 07:23:29 +00:00
Antonio Andelic
4aa8c5fe49 Merge branch 'master' into alternative-keeper-configs 2023-03-28 07:23:24 +00:00
Raúl Marín
ff209db129
Merge branch 'master' into zk_inject_timeout 2023-03-27 19:46:10 +02:00
Sema Checherinda
a6ab33a906 no use query, but storage context 2023-03-27 16:30:32 +02:00
Han Fei
e3afa5090f
Merge pull request #47218 from hanfei1991/hanfei/optimize-regexp-tree-1 
Refine OptimizeRegularExpression Function and RegexpTreeDict
 2023-03-27 15:23:01 +02:00
Raúl Marín
714b54b322 Reimplement enable_fault_injections_during_startup 2023-03-27 15:06:46 +02:00
Sema Checherinda
26e76bb68c
Merge branch 'master' into fix_JSON_searchField 2023-03-27 10:47:57 +02:00
Sema Checherinda
db55fa864a fix build 2023-03-27 10:47:34 +02:00
Antonio Andelic
93f5920e02
Merge branch 'master' into alternative-keeper-configs 2023-03-27 10:00:21 +02:00
Vasily Nemkov
fa27ed9d41 Less noisy tests 2023-03-25 23:31:17 +01:00
Vasily Nemkov
ce608b135f Fixed implementation of find_first_symbols_sse42 run-time needle 
Added some tests
 2023-03-25 23:18:51 +01:00
Kruglov Pavel
713ecb2492
Merge branch 'master' into non-blocking-connect 2023-03-24 22:55:45 +01:00
Sema Checherinda
da4f2bd923 do not attach empty thread group 2023-03-24 19:53:16 +01:00
Sema Checherinda
bc107c70fa merge and mutation make thread group for setting memory trackers right 2023-03-24 19:53:16 +01:00
Maksim Kita
5cb2d30563
Merge pull request #47316 from ClickHouse/vdimir/transform_external_query_analyzer 
Support transformQueryForExternalDatabase for analyzer
 2023-03-24 17:53:00 +01:00
Raúl Marín
bf7b0c5cfe Different approach to inject timeouts 2023-03-24 17:12:14 +01:00
Sema Checherinda
1e462e02ed add ut 2023-03-24 17:00:04 +01:00
Sergei Trifonov
a05aa5c1c6
Merge branch 'master' into segmentator-fix 2023-03-24 16:19:57 +01:00
Kruglov Pavel
f3f93dd06c
Merge branch 'master' into non-blocking-connect 2023-03-24 15:59:40 +01:00
serxa
28ca2de86d fix 2023-03-24 14:58:50 +00:00
Vasily Nemkov
e22eee8055 Updated implementation of find_first_symbols for run-time needle 
Implemented compile-time and run-time dispatching between SSE4.2 and SSE2 implementations
Added find_first_symbols_sse2
Added tests
 2023-03-24 15:45:12 +01:00
Kruglov Pavel
3ee12e21fb
Merge branch 'master' into non-blocking-connect 2023-03-23 20:53:44 +01:00
Antonio Andelic
3ae09c6ce8
Merge branch 'master' into more-s3-events 2023-03-23 18:23:53 +01:00
Antonio Andelic
d6cbc5d05b Fix tests 2023-03-23 15:01:51 +00:00
Antonio Andelic
9ea8dc4e98 Merge branch 'master' into alternative-keeper-configs 2023-03-23 09:22:46 +00:00
Antonio Andelic
b57e82ded6 Add S3 write profile events 2023-03-23 09:05:41 +00:00
Antonio Andelic
a4d1cd514d Refactor 2023-03-23 08:58:56 +00:00
Han Fei
02de4ad6df address comments 2023-03-22 17:50:19 +01:00
Han Fei
575c4263a3 address comments 2023-03-22 17:47:25 +01:00
Yatsishin Ilya
5786aba5eb Merge remote-tracking branch 'origin/master' into clickhouse-help 2023-03-22 14:08:22 +00:00
serxa
4321569fc4 Merge branch 'segmentator-fix' of github.com:ClickHouse/ClickHouse into segmentator-fix 2023-03-22 13:51:23 +00:00
serxa
f3c12b8543 fix per-thread perf counters update 2023-03-22 13:51:07 +00:00
Antonio Andelic
65b99ec6a7 Merge branch 'master' into alternative-keeper-configs 2023-03-22 12:22:57 +00:00
vdimir
023d14a894 Support transformQueryForExternalDatabase for analyzer 2023-03-22 08:59:04 +01:00
Vitaly Baranov
4318212c73 Make a condition more clear. 2023-03-21 21:52:59 +01:00
Vitaly Baranov
096d0132af More corrections. 2023-03-21 21:52:59 +01:00
Vitaly Baranov
0a0e9d060f Corrections after review. 2023-03-21 21:52:59 +01:00
Vitaly Baranov
85d38493f6 Improve ThreadPool 
ThreadPool::setMaxThreads() can now start new threads (if there are scheduled jobs in the queue for them);
ThreadPool::setMaxFreeThreads() can now finish free threads.
 2023-03-21 21:52:59 +01:00
Nikolay Degterinsky
3a8314c13e
Merge pull request #47576 from HarryLeeIBM/hlee-s390x-storage-gen 
Fix SipHash hashing and byte order in GenerateRandom for s389x
 2023-03-21 21:06:08 +01:00
Arthur Passos
77419db791 add a few tests for runtime find_symbols, more are needed 2023-03-21 09:07:06 -03:00
Kseniia Sumarokova
3c550b4314
Merge pull request #46647 from kssenii/named-collections-finish 
Named collections: finish replacing old code for storages
 2023-03-21 12:36:46 +01:00
Kseniia Sumarokova
a9bcd022d5
Merge pull request #46241 from kssenii/named-collections-granular-access 
Allow separate grants for every named collection
 2023-03-21 12:27:53 +01:00
Vladimir C
5add1e1b2e
Merge pull request #47684 from ClickHouse/vdimir/improve_log_queries_cut 2023-03-21 11:17:54 +01:00
Robert Schulze
3b7f929001
Merge branch 'master' into rs/qc-cachebase 2023-03-20 17:08:47 +01:00
vdimir
8c48714236
wipeSensitiveDataAndCutToLength take into accound suffix length 2023-03-20 15:54:19 +00:00
kssenii
75624c52f7 Merge remote-tracking branch 'upstream/master' into named-collections-granular-access 2023-03-20 16:38:14 +01:00
vdimir
bb7a8ed862
Apply log_queries_cut_to_length in MergeTreeWhereOptimizer 2023-03-20 15:38:09 +00:00
Vitaly Baranov
198409e12a
Merge pull request #46085 from aalexfvk/alexfvk/store_udf_in_zookeeper 
Replication of user-defined SQL functions using ZooKeeper
 2023-03-20 13:41:29 +01:00
Kruglov Pavel
1d12e85d55
Merge branch 'master' into non-blocking-connect 2023-03-20 13:23:55 +01:00
kssenii
cae3b335d6 Merge remote-tracking branch 'upstream/master' into named-collections-finish 2023-03-20 11:23:22 +01:00
Alexey Milovidov
698fcba525
Merge branch 'master' into fix-mysql-leak 2023-03-20 07:37:01 +03:00
Alexey Milovidov
ddf29b69f9
Merge branch 'master' into fuzzer-of-data-formats 2023-03-20 03:26:51 +03:00
kssenii
ea375ef989 Fix memory leak 2023-03-20 00:40:06 +01:00
Robert Schulze
0e01e912fe
Preparation for libcxx(abi), llvm, clang-tidy 16 2023-03-19 17:25:41 +00:00
Robert Schulze
c4664c70a1
Merge branch 'master' into rs/qc-cachebase 2023-03-19 15:23:24 +01:00
Alexey Milovidov
6e1b45d1d6 Fix exotic trash 2023-03-19 06:48:18 +01:00
Alexey Milovidov
a99f10e16f Better error messages 2023-03-19 05:34:44 +01:00
Alexey Milovidov
ef01579726 Fix trash - the code written in C with pointers 2023-03-19 03:19:15 +01:00
Alexander Tokmakov
f1df91cd09
Merge pull request #47659 from ClickHouse/fix_993 
Fixes for 993 and friends
 2023-03-18 14:46:43 +03:00
Sema Checherinda
0b641fcead
Merge pull request #47564 from CheSema/remove-redundant 
remove counters for threads, fix negative counters
 2023-03-18 02:23:52 +01:00
Alexander Tokmakov
d11aee2b0c
Merge branch 'master' into fix_993 2023-03-17 21:28:23 +03:00
Sema Checherinda
e8295be9be if no group then do attachInternalTextLogsQueue to the thread only 2023-03-17 15:56:30 +01:00
Robert Schulze
f72a337074
Remove cruft from build 
No need to check compiler flags, clang >= 15 supports all of them.
 2023-03-17 13:44:04 +00:00
kssenii
bb0beb7449 Merge remote-tracking branch 'upstream/master' into named-collections-finish 2023-03-17 13:02:36 +01:00
Sema Checherinda
e2af8f6ed6 add a comment, rename a field 2023-03-17 11:59:44 +01:00
Aleksei Filatov
886b530963 [rev: 1] Fix review remarks 2023-03-17 13:56:05 +03:00
Aleksei Filatov
f84aa2f4ca Add classes for replication UDF to ZooKeeper 2023-03-17 13:56:04 +03:00
Sergei Trifonov
1a84203c7f
Merge pull request #47656 from ClickHouse/background_pool_limit_metric 
Add background pools size metrics
 2023-03-17 11:49:53 +01:00
Alexander Tokmakov
d621b2c4ad fix intersecting parts, better fault injections 2023-03-16 21:28:07 +01:00
Sema Checherinda
3c6deddd1d work with comments on PR 2023-03-16 19:55:58 +01:00
serxa
3ad7194e31 add pool size metrics 2023-03-16 17:31:41 +00:00
Robert Schulze
eb206bbe25
Fix clang-tidy build 2023-03-16 16:08:24 +00:00
Robert Schulze
7d7052350c
Merge branch 'master' into rs/qc-cachebase 2023-03-16 10:42:17 +01:00
sichenzhao
88de8d4cb3
Merge branch 'master' into cgroup_v2 2023-03-15 14:35:47 -07:00
alesapin
ef7508469c Fix wait for zero-copy replication move during manual move 2023-03-15 22:29:24 +01:00
Sema Checherinda
67ac858a52 remove counters for threads, fix negative counters 2023-03-15 22:22:22 +01:00
Dmitry Novik
be40218a9a
Revert "Revert #46622 (test_async_insert_memory)" 2023-03-15 22:14:54 +01:00
Han Fei
a532503466
Merge branch 'master' into hanfei/optimize-regexp-tree-1 2023-03-15 17:56:01 +01:00
Han Fei
d78a9e03ad refine 2023-03-15 15:38:11 +01:00
avogar
17f5356178 Improve async reading from socket 2023-03-15 12:18:08 +00:00
robot-ch-test-poll1
43db7c59d9
Merge pull request #47260 from ClickHouse/vdimir/storage_join_insert_select_deadlock 
Forbid insert select for the same StorageJoin
 2023-03-15 12:52:29 +01:00
Robert Schulze
2f75a6da61
Merge remote-tracking branch 'origin/master' into rs/qc-cachebase 2023-03-15 11:00:53 +00:00
Robert Schulze
75ca2dd0e2
Merge remote-tracking branch 'origin/master' into rs/qc-cachebase 2023-03-15 10:55:53 +00:00
Robert Schulze
5cee479f0c
Merge pull request #47424 from ClickHouse/rs/compile-with-cpp23 
Compile with C++23
 2023-03-15 10:55:42 +01:00
Yatsishin Ilya
45c5275bfb Merge remote-tracking branch 'origin/master' into clickhouse-help 2023-03-14 17:27:25 +00:00
Han Fei
076d33bb03 refine a little bit 2023-03-14 18:15:42 +01:00
Han Fei
01be209e43 fix test 2023-03-14 17:44:02 +01:00
HarryLeeIBM
a32d7a00bc Fix SipHash integer hashing and byte order issue in GenerateRandom for s390x 2023-03-14 11:54:17 -04:00
Sema Checherinda
95e994a9cd
Merge pull request #47304 from arthurpassos/fix_find_first_not_symbols 
Modify find_first_symbols so it works as expected for find_first_not_symbols
 2023-03-14 14:24:05 +01:00
Alexander Tokmakov
5a1ee881f8
Merge pull request #47547 from ClickHouse/filter_too_frequent_logs 
Hide too noisy log messages, fix some tests
 2023-03-14 13:53:21 +03:00
Robert Schulze
73afae2d3f
Introduce TTLCachePolicy 2023-03-14 09:45:20 +00:00
Robert Schulze
eed365c8c8
Cleanup: Slightly more naming consistency (probably rolls back earlier changes) 2023-03-14 09:11:36 +00:00
Robert Schulze
b985795593
Cleanup: Group ICachePolicy methods 2023-03-14 08:38:22 +00:00
Robert Schulze
e1fb25a004
Extend CacheBase with getWithKey() method 2023-03-14 08:38:22 +00:00
Robert Schulze
f1450923cb
Extend CacheBase with dump() method 2023-03-14 08:38:22 +00:00
Robert Schulze
e03618357e
Cleanup: Acquire lock in CacheBase::maxSize() 
The upcoming TTLPolicy will likely have a runtime-configurable cache
size.
 2023-03-14 08:38:22 +00:00
Michael Kolupaev
d3a514d221 Compress marks in memory 2023-03-13 16:29:00 -07:00
Alexander Tokmakov
ba6c6255a3 hide too noisy log messages, fix some tests 2023-03-14 00:23:36 +01:00
Alexey Milovidov
a411ae10b5
Merge pull request #47486 from ClickHouse/revert_25674 
Remove "asynchronous connection drain"
 2023-03-13 21:40:39 +03:00
Han Fei
de8d0040a4 refine code 2023-03-13 18:34:47 +01:00