thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-16 19:32:07 +00:00
Code Issues Packages Projects Releases Wiki Activity
112,483 Commits 9,319 Branches 1,783 Tags 2.1 GiB
d1793a9bc5
Commit Graph

39835 Commits

Author SHA1 Message Date
Sema Checherinda
0503ed6fb8
Merge pull request #46668 from azat/fix-query_views_log-from-bg 
Fix system.query_views_log for MVs that are pushed from background threads
 2023-04-07 14:47:05 +02:00
Igor Nikonov
78038a3c2c Fix: do not resize pipeline when there is no files to process (globs expands to empty set) 2023-04-07 11:34:04 +00:00
MikhailBurdukov
54d7a412ec Fixed unit tests 2023-04-07 11:04:00 +00:00
Nikolay Degterinsky
a48655af39
Merge branch 'master' into add-kafka-murmur-hash-function 2023-04-07 12:21:16 +02:00
robot-ch-test-poll2
d13e8f8ddb
Merge pull request #48518 from ClibMouse/feature/remove-dead-code 
Remove dead code and unused dependencies
 2023-04-07 12:07:33 +02:00
alesapin
a0dd1c96bf
Merge pull request #48512 from ClickHouse/serxa-patch-3 
Fix bytesSize() of zk SetRequest
 2023-04-07 11:47:31 +02:00
Kseniia Sumarokova
d5889a47d6
Merge pull request #48311 from kssenii/rabbitmq-fix-cpu-usage 
Fix cpu usage in rabbitmq (was worsened in 23.2 after #44404)
 2023-04-07 11:44:48 +02:00
Azat Khuzhin
011480924a Use forward declaration of ThreadPool 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-07 11:25:35 +02:00
Antonio Andelic
67e6335250 Fix nullptr to memcpy 2023-04-07 08:32:01 +00:00
Sergei Trifonov
f0de79b21a
Merge branch 'master' into lost-part-count 2023-04-07 01:50:41 +02:00
Yakov Olkhovskiy
8d1924cc9a
Merge pull request #47962 from ClickHouse/imp-use-structure-from-insertion 
Improvement and fix for use_structure_from_insertion_table_in_table_functions
 2023-04-06 19:01:00 -04:00
serxa
343a071798 add lost_part_count column to system.replicas 2023-04-06 22:38:58 +00:00
Igor Nikonov
96213fa464 Fix header 2023-04-06 22:17:09 +00:00
Igor Nikonov
2e139c21d2 Parallel reading in FROM file() 2023-04-06 21:57:03 +00:00
robot-clickhouse-ci-2
2cb8252676
Merge pull request #48487 from azat/memory-tracker-fix-deadlock 
Fix deadlock due to debug tracking of memory allocations
 2023-04-06 23:50:44 +02:00
Dmitry Novik
a1552159e3 Fix style 2023-04-06 21:35:28 +00:00
ltrk2
4449e49e72 Mark a function inline 2023-04-06 13:28:30 -07:00
Nikolai Kochetov
57255eb56c Fix equals. 2023-04-06 19:22:10 +00:00
Dmitry Novik
a60c315b3e Support additional_result_filter 2023-04-06 19:18:26 +00:00
Nikolai Kochetov
e8fb135180 Reduce memory usage for multiple alter delete mutations. 2023-04-06 19:07:53 +00:00
ltrk2
790e570af6 Use std::string::starts_with instead of a roll your own variant 2023-04-06 11:57:44 -07:00
ltrk2
00c8e6e60d Use std::string_view instead of strlen 2023-04-06 11:39:33 -07:00
ltrk2
4544abc7d6 Remove dead code and unused dependencies 2023-04-06 11:37:12 -07:00
Sergei Trifonov
b6975d36e9
Fix bytesSize() of zk SetRequest 2023-04-06 19:05:32 +02:00
Sema Checherinda
9d8d9e8ec9
Merge branch 'master' into fix-query_views_log-from-bg 2023-04-06 18:16:47 +02:00
Anton Popov
4757d76fbf fix query cache with sparse columns 2023-04-06 16:01:23 +00:00
MikhailBurdukov
5282a20aba Test and fix 2023-04-06 15:50:28 +00:00
Alexey Milovidov
4ab8b35b00
Merge pull request #48444 from ClickHouse/better-exception-messages-from-keeper-client 
Better exception messages from Keeper client
 2023-04-06 18:36:39 +03:00
Alexey Milovidov
3066d5df52
Merge pull request #48195 from ClickHouse/disable-compile-aggregations 
Remove a feature
 2023-04-06 18:09:48 +03:00
Anton Popov
b5a3723c88
Merge pull request #48395 from ClickHouse/with_fill_clarification 
WITH FILL clarification and cleanup
 2023-04-06 17:02:57 +02:00
Robert Schulze
7bcb8c4970
Register trim aliases in system.functions 2023-04-06 14:25:27 +00:00
Anton Popov
e88938d6c8
Change error code. 2023-04-06 16:17:45 +02:00
Robert Schulze
cfd9c4d85e
Register aliases of date_diff in system.functions 2023-04-06 14:04:37 +00:00
Azat Khuzhin
e91c8bfea3 Fix deadlock due to debug tracking of memory allocations 
Since memory tracker should be blocked globally (by default it is blocked only
for VariableContext::User).

<details>

<summary>stacktrace</summary>

    * thread 11, name = 'Collector', stop reason = trace
        frame 1: 0x00007ffff7e37ea2 libc.so.6`pthread_mutex_lock + 274
        frame 6: 0x000000001ee10298 clickhouse`toStringCached(pointers=0x00007fffedfef068, offset=0, size=45) + 88 at StackTrace.cpp:425
        frame 7: 0x000000001ee1128e clickhouse`StackTrace::toString(this=0x00007fffedfef058) const + 46 at StackTrace.cpp:444
        frame 8: 0x000000001ede58ff clickhouse`(anonymous namespace)::debugLogBigAllocationWithoutCheck(size=25187648) + 479 at MemoryTracker.cpp:98
        frame 9: 0x000000001ede5481 clickhouse`MemoryTracker::allocImpl(this=0x0000000030729e68, size=25187648, throw_if_memory_exceeded=false, query_tracker=0x00007ffff3c22440) + 3937 at MemoryTracker.cpp:352
        frame 10: 0x000000001ede477f clickhouse`MemoryTracker::allocImpl(this=0x00007ffff3c1e530, size=25187648, throw_if_memory_exceeded=false, query_tracker=0x00007ffff3c22440) + 607 at MemoryTracker.cpp:198
        frame 11: 0x000000001ede477f clickhouse`MemoryTracker::allocImpl(this=0x00007ffff3c22440, size=25187648, throw_if_memory_exceeded=false, query_tracker=0x0000000000000000) + 607 at MemoryTracker.cpp:198
        frame 12: 0x000000001ede477f clickhouse`MemoryTracker::allocImpl(this=0x00007fffedff3bf0, size=25187648, throw_if_memory_exceeded=false, query_tracker=0x0000000000000000) + 607 at MemoryTracker.cpp:198
        frame 13: 0x000000001ed6c4c9 clickhouse`CurrentMemoryTracker::allocImpl(size=25165824, throw_if_memory_exceeded=false) + 265 at CurrentMemoryTracker.cpp:58
        frame 14: 0x000000001ed6c6a1 clickhouse`CurrentMemoryTracker::allocNoThrow(size=25165824) + 33 at CurrentMemoryTracker.cpp:91
        frame 15: 0x000000001ed63079 clickhouse`operator new(unsigned long) [inlined] void Memory::trackMemory<>(size=25165824) + 89 at memory.h:177
        frame 16: 0x000000001ed63034 clickhouse`operator new(size=25165824) + 20 at new_delete.cpp:74
        ...
        frame 27: 0x000000001ee15895 clickhouse`DB::(anonymous namespace)::collectSymbols(info=0x00007ffff7062000, (null)=<unavailable>, data_ptr=0x00007fffec2b7000) + 53 at SymbolIndex.cpp:483
        frame 28: 0x000000002e1a5d57 clickhouse`::dl_iterate_phdr(callback=(clickhouse`DB::(anonymous namespace)::collectSymbols(dl_phdr_info*, unsigned long, void*) at SymbolIndex.cpp:480), data=0x00007fffec2b7000)(dl_phdr_info *, size_t, void *), void *) + 183 at phdr_cache.cpp:76
        frame 29: 0x000000001ee15549 clickhouse`DB::SymbolIndex::update(this=0x00007fffec2b7000) + 41 at SymbolIndex.cpp:515
        frame 30: 0x000000001ee1be35 clickhouse`DB::SymbolIndex::SymbolIndex(this=<unavailable>) + 53 at SymbolIndex.h:23
        frame 31: 0x000000001ee17a6c clickhouse`DB::SymbolIndex::instanceImpl() + 108 at SymbolIndex.cpp:555
        frame 32: 0x000000001ee17b4a clickhouse`DB::SymbolIndex::instance() + 10 at SymbolIndex.cpp:561
        frame 33: 0x000000001ee1070f clickhouse`void toStringEveryLineImpl<toStringCached(std::__1::array<void*, 45ul> const&, unsigned long, unsigned long)::$_0>(fatal=false, stack_trace=0x00007fffedff1ef8, callback=0x00007fffedff1ee0)::$_0&&) + 207 at StackTrace.cpp:349
        frame 34: 0x000000001ee1040b clickhouse`toStringCached(pointers=0x00007fffedff2928, offset=0, size=43) + 459 at StackTrace.cpp:436
        frame 35: 0x000000001ee1128e clickhouse`StackTrace::toString(this=0x00007fffedff2918) const + 46 at StackTrace.cpp:444
        frame 36: 0x000000001ede58ff clickhouse`(anonymous namespace)::debugLogBigAllocationWithoutCheck(size=25165824) + 479 at MemoryTracker.cpp:98
        frame 37: 0x000000001ede5481 clickhouse`MemoryTracker::allocImpl(this=0x00007fffedff3bf0, size=25165824, throw_if_memory_exceeded=false, query_tracker=0x0000000000000000) + 3937 at MemoryTracker.cpp:352
        frame 38: 0x000000001ed6c4c9 clickhouse`CurrentMemoryTracker::allocImpl(size=25165824, throw_if_memory_exceeded=false) + 265 at CurrentMemoryTracker.cpp:58
        frame 39: 0x000000001ed6c6a1 clickhouse`CurrentMemoryTracker::allocNoThrow(size=25165824) + 33 at CurrentMemoryTracker.cpp:91
        frame 40: 0x000000001ed63079 clickhouse`operator new(unsigned long) [inlined] void Memory::trackMemory<>(size=25165824) + 89 at memory.h:177
        frame 41: 0x000000001ed63034 clickhouse`operator new(size=25165824) + 20 at new_delete.cpp:74
        ...
        frame 55: 0x0000000025badace clickhouse`DB::WriteBufferFromS3::nextImpl(this=0x00007ffff3c58f00) + 366 at WriteBufferFromS3.cpp:110

</details>

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-06 16:03:46 +02:00
Sema Checherinda
b8729be97a
Merge pull request #47104 from CheSema/thead-group-for-merge 
merge and mutation make thread group for setting memory trackers right
 2023-04-06 15:46:56 +02:00
kssenii
2d335f82cc Fix another test 2023-04-06 15:43:16 +02:00
Vitaly Baranov
b45b662fe3
Merge pull request #48431 from Algunenano/system_referential_table 
Don't check dependencies when renaming system tables automatically
 2023-04-06 15:28:37 +02:00
Nikolai Kochetov
6c6335b686
Merge pull request #48357 from ClickHouse/fix-projections-analysis-with-analyzer 
Batch fix for projections analysis with analyzer.
 2023-04-06 15:13:56 +02:00
Nikolai Kochetov
54180851ab Fix name 2023-04-06 13:13:17 +00:00
Yakov Olkhovskiy
d2e897d4b8
Merge branch 'master' into imp-use-structure-from-insertion 2023-04-06 08:52:38 -04:00
Nikolai Kochetov
b199219ec1 Add reading step for system zookeeper. Analyze path from filter DAG. 2023-04-06 12:24:22 +00:00
Vitaly Baranov
2fc40834f5
Merge branch 'master' into fix-forgotten-showsecrets-in-semi-copy-ctor 2023-04-06 14:20:57 +02:00
Nikolai Kochetov
e04abef6b3
Merge pull request #48469 from ClickHouse/kssenii-patch-7 
Fix possible segfault in cache
 2023-04-06 13:56:45 +02:00
Kruglov Pavel
fb33d6bedb
Merge pull request #48126 from Avogar/orc-arrow-parquet-more-types 
Support more ClickHouse types in ORC/Arrow/Parquet formats
 2023-04-06 13:32:46 +02:00
Kruglov Pavel
1ec9f5e42c
Avoid operation on uninitialised data in readDateTimeTextImpl 2023-04-06 13:20:11 +02:00
Kseniia Sumarokova
3fc952a56a
Update CachedOnDiskReadBufferFromFile.cpp 2023-04-06 13:01:24 +02:00
Antonio Andelic
28a33f0a47
Merge pull request #48435 from ClickHouse/kv-update-only-affected-rows 
Update only affected rows in KV storage
 2023-04-06 12:33:54 +02:00
Robert Schulze
df5de4ec78
Merge remote-tracking branch 'origin/master' into rs/better-f-in-formatdatetime 2023-04-06 09:58:09 +00:00
Vladimir C
e7a5c96353
Merge pull request #48219 from ClickHouse/vdimir/lock_order_47072 2023-04-06 11:54:33 +02:00
Vladimir C
a2b71a0054
Merge pull request #48386 from ClickHouse/vdimir/reset-downloader-tmp-data-cache 2023-04-06 11:23:16 +02:00
Robert Schulze
b7be5fd89e
Improve code and tests 2023-04-06 08:47:38 +00:00
Robert Schulze
5eb31bba87
MySQL compat: Align behavior of %f in formatDateTime() with parseDateTime() 2023-04-06 08:13:04 +00:00
Antonio Andelic
a52fc83e14
Merge branch 'master' into dont-replicate-more-keepermap 2023-04-06 09:35:37 +02:00
Robert Schulze
ce2692ba79
Merge pull request #47246 from ClickHouse/rs/m 
Change the behavior of formatter %M in function formatDateTime() from minutes to month name
 2023-04-06 09:19:50 +02:00
Yakov Olkhovskiy
8a19ac4e73
Merge branch 'master' into imp-use-structure-from-insertion 2023-04-06 00:51:44 -04:00
Nikolay Degterinsky
95086536b7 Fix style 2023-04-06 01:22:39 +00:00
Anton Popov
166a63c36c
Merge pull request #46589 from ClickHouse/rs/nonconst-replace 
replace[Regexp]{One,All}(): allow non-const pattern and replacement arguments
 2023-04-06 03:08:28 +02:00
Nikolay Degterinsky
0dfcc84910 Add support for Date/Datetime datatypes to arrayMin, arrayMax, arrayDifference functions 2023-04-06 00:53:53 +00:00
Alexey Milovidov
785ea8213b Better exception messages from Keeper client 2023-04-06 02:29:37 +02:00
robot-ch-test-poll3
3ad0a6ac18
Merge pull request #48433 from ClickHouse/some-fixes-for-parallel-replicas 
Some fixes for parallel replicas
 2023-04-06 02:00:06 +02:00
robot-ch-test-poll1
9466cec1fc
Merge pull request #48342 from ClickHouse/Backup_Restore_concurrency_check_node_2 
Check node for Backup Restore concurrency
 2023-04-05 23:49:32 +02:00
Anton Popov
4fb7e0ec3c add retries to loading of data parts 2023-04-05 21:23:57 +00:00
Alexander Tokmakov
a154fa73f4
Merge pull request #48401 from ClickHouse/fix_00002_log_messages 
Find big allocations without memory limits checks
 2023-04-05 23:52:39 +03:00
Robert Schulze
65609340cc
Yet another fix 2023-04-05 20:23:41 +00:00
robot-ch-test-poll2
bf003c7595
Merge pull request #48390 from Avogar/protobuf-tuple 
Allow write/read unnamed tuple as nested Message in Protobuf format
 2023-04-05 22:14:28 +02:00
Robert Schulze
3e84be5f3a
Fix stuff 2023-04-05 20:10:27 +00:00
Robert Schulze
ce234baf9c
Merge branch 'master' into rs/f 2023-04-05 21:59:13 +02:00
Robert Schulze
809a7fbb7a
Incorporate more review feedback 2023-04-05 19:43:30 +00:00
Sema Checherinda
554c928534
Merge branch 'master' into thead-group-for-merge 2023-04-05 21:09:38 +02:00
Raúl Marín
1fa992f015
Merge branch 'master' into system_referential_table 2023-04-05 20:47:53 +02:00
Nikolay Degterinsky
061be7c9ad Add trailing commas to expression list in SELECT query 2023-04-05 18:08:56 +00:00
Sema Checherinda
c1adbb7aa6 transfer counters to the query directly 2023-04-05 19:51:36 +02:00
robot-clickhouse
3d2c279859
Merge pull request #48242 from azat/better-throttling 
Per-query/per-server throttling for remote IO/local IO/BACKUPs
 2023-04-05 19:32:37 +02:00
Robert Schulze
90a1a75424
Merge pull request #48412 from ClickHouse/rs/fix-ast-fuzzer 
AST fuzzer: Fix assertion in TopK serialization
 2023-04-05 19:16:50 +02:00
Kruglov Pavel
11d3d01e50
Fix special build 2023-04-05 18:29:59 +02:00
Kruglov Pavel
faf6d49f1b
Merge pull request #48356 from Avogar/fix-protobuf-abort 
Avoid abort in protobuf library in debug build
 2023-04-05 18:26:59 +02:00
Kruglov Pavel
9331c6c260
Merge pull request #48122 from Avogar/bson-more-types 
Support Enum output/input in BSONEachRow, allow all map key types and avoid extra calculations
 2023-04-05 18:26:19 +02:00
Nikolai Kochetov
ab87cf7ff7
Merge branch 'master' into fix-possible-pipeline-stuck-on-partial-query-result 2023-04-05 18:25:09 +02:00
avogar
1be6bceb2f Generate valid Decimals and Bools in generateRandom function 2023-04-05 16:23:06 +00:00
Vitaly Baranov
82373890e5
Merge branch 'master' into fix-forgotten-showsecrets-in-semi-copy-ctor 2023-04-05 18:17:33 +02:00
Dmitry Novik
4cb1ffa2eb Fix distributed case 2023-04-05 16:13:03 +00:00
Antonio Andelic
0260b84bc3 Update only affected rows in KV storage 2023-04-05 16:12:14 +00:00
Raúl Marín
5d9e006c46 PR comments 2023-04-05 18:04:03 +02:00
Igor Nikonov
46f7016d8c Add search alias by input name in ActionsDAG 2023-04-05 15:56:06 +00:00
Antonio Andelic
7975df538e Don't replicate KeeperMap DELETE and TRUNCATE 2023-04-05 15:28:35 +00:00
Nikita Mikhaylov
3f44c6d936 Better 2023-04-05 15:19:45 +00:00
Alexey Milovidov
5477d67acd
Merge branch 'master' into disable-compile-aggregations 2023-04-05 18:17:17 +03:00
kssenii
62244ee57d Fix test 2023-04-05 17:14:41 +02:00
Vitaly Baranov
862a4d6fcd
Merge pull request #48142 from DimasKovas/json_functions_to_h_file 
Move FunctionsJSON implementation to header file
 2023-04-05 16:52:48 +02:00
Anton Popov
027f8a02c8
Merge branch 'master' into check_sample_dict_key_is_correct 2023-04-05 16:31:57 +02:00
Raúl Marín
d65d00f496 Don't check dependencies when renaming system tables automatically 2023-04-05 16:15:24 +02:00
vdimir
d9abcf2f69
Reduce scope storage_policies_mutex in setTemporaryStoragePolicy 2023-04-05 13:42:49 +00:00
vdimir
01ce3baf0f
Rename Context::getTemporaryVolume to getGlobalTemporaryVolume 2023-04-05 13:42:13 +00:00
Igor Nikonov
41f6e0e7b5 More efficient original node search in ActionsDAG 2023-04-05 13:36:07 +00:00
vdimir
6f3fa33641
Add unbound flag to system.filesystem_cache 2023-04-05 13:12:20 +00:00
Alexander Tokmakov
0e0e89d596 fix 2023-04-05 15:10:09 +02:00
Robert Schulze
62608f74d3
Merge pull request #48396 from ClickHouse/rs/clang-tidy-cleanup 
Cleanup mess in .clang-tidy
 2023-04-05 14:39:28 +02:00
Alexander Tokmakov
3b5ed9242c
Merge pull request #48391 from azat/fix-sync-replica-stuck 
Fix possible SYSTEM SYNC REPLICA stuck in case of DROP/REPLACE PARTITION
 2023-04-05 15:20:46 +03:00
Robert Schulze
949f7c1839
Fix darwin build 2023-04-05 12:07:22 +00:00
Kruglov Pavel
bd318950b3
Fix special build 2023-04-05 13:35:12 +02:00
Nikita Mikhaylov
7b526d3df9
Always communicate with local replica through loopback interface for parallel replicas (#48246) 2023-04-05 13:32:46 +02:00
avogar
9d3790a4c8 Fix tests 2023-04-05 11:27:38 +00:00
Robert Schulze
1be81db885
Fix build, pt. II 2023-04-05 11:23:09 +00:00
Robert Schulze
a59b53775c
Fix style 2023-04-05 10:43:55 +00:00
Sema Checherinda
f504cd5fed avoid race, do not set thread_group.counters.parent from query 2023-04-05 12:36:55 +02:00
Igor Nikonov
f6c46153fb Rename old_*_columns to input_*_columns 2023-04-05 10:22:26 +00:00
Igor Nikonov
47e7e19ca9 Move suffix generation code into one function 2023-04-05 10:12:42 +00:00
vdimir
bae1286b63
Update temp_data_on_disk in shared context 2023-04-05 10:07:44 +00:00
Kruglov Pavel
96a3307bda
Merge branch 'master' into fix-protobuf-abort 2023-04-05 11:57:18 +02:00
Kruglov Pavel
8d6b709f1a
Merge pull request #48257 from Avogar/capnp-more-types 
Support more types in CapnProto format, allow integer conversions
 2023-04-05 11:56:46 +02:00
avogar
f46f098c78 Better 2023-04-05 09:55:49 +00:00
Vladimir C
1f22fca3c3
Merge pull request #46428 from bigo-sg/quantile_gk 2023-04-05 11:25:48 +02:00
vdimir
7fe06f8435
Do not take lock for shared context in setTempDataOnDisk 2023-04-05 09:22:09 +00:00
Raúl Marín
2276d4feb4 Backups have no context and no process list element 2023-04-05 11:19:04 +02:00
Robert Schulze
0d5d2a9b55
Implement %f in parseDateTime() 
Fixes: #48394

@cc OP
 2023-04-05 09:13:18 +00:00
Azat Khuzhin
4008155a6e Add max_ prefix for backup_bandwidth settings 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-05 10:54:25 +02:00
Robert Schulze
598b050628
AST Fuzzer: Fix assertion in TopK serialization 
Problem:
  https://s3.amazonaws.com/clickhouse-test-reports/0/fa5b2bd4a5b02336bca8837c473a7124f8ecedf2/fuzzer_astfuzzerasan/report.html

The new assertion in the Varint code was introduced with (*). It rejects
values whose serialization cannot be deserialized (and this behavior
cannot be changed due to historical reasons). Such values should be
exceptionally rare in practice but AST fuzzer managers to trigger them.

The fix is similar to (**): Bypass the check by limiting the value to
the maximum allowed value.

(if AST fuzzer triggers finds more violations of the assertion, we might
consider throwing an exception instead)

(*) https://github.com/ClickHouse/ClickHouse/pull/48154
(**) https://github.com/ClickHouse/ClickHouse/pull/48154/files#diff-653c0a18dfdaa86262c78dc6b25550add0487f165b4ad053e86f530388f6203a
 2023-04-05 08:07:37 +00:00
Azat Khuzhin
f0f58de79c Remove throttler for IDisk::copyFile() 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-05 09:41:27 +02:00
Azat Khuzhin
dd9f0f409b Remove knowledge about throttling from IBackupWriter::supportNativeCopy() 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-05 09:40:19 +02:00
Azat Khuzhin
c8597fbb9a Do not throttle S3-S3 backups if native copy is possible 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-05 09:40:18 +02:00
Azat Khuzhin
61405b827d Use MAKE_DEPRECATED_BY_SERVER_CONFIG() for deprecated throttle settings 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-05 09:40:18 +02:00
Azat Khuzhin
684ecc24e2 Ignore deprecated values max_remote_{read,write}_network_bandwidth_for_server 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-05 09:40:18 +02:00
Azat Khuzhin
ad96d3c249 Do not require proper Poco Application for Context 
$ yes | head | clickhouse-obfuscator --structure 'id String' --seed foo --input-format TSV --output-format TSV
    Poco::Exception. Code: 1000, e.code() = 0, Null pointer: _pInstance in file "/src/ch/clickhouse/base/poco/Util/include/Poco/Util/Application.h", line 446, Stack trace (when copying this message, always include the lines below):

    0. /src/ch/clickhouse/contrib/llvm-project/libcxx/include/exception:134: std::exception::capture() @ 0x15605622 in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
    1. /src/ch/clickhouse/contrib/llvm-project/libcxx/include/exception:112: std::exception::exception[abi:v15000]() @ 0x156055ed in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
    2. /src/ch/clickhouse/base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x2ebd2d80 in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
    3. /src/ch/clickhouse/base/poco/Foundation/src/Exception.cpp:132: Poco::LogicException::LogicException(String const&, int) @ 0x2ebd3667 in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
    4. /src/ch/clickhouse/base/poco/Foundation/src/Exception.cpp:134: Poco::NullPointerException::NullPointerException(String const&, int) @ 0x2ebd3da7 in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
    5. /src/ch/clickhouse/base/poco/Foundation/src/Bugcheck.cpp:42: Poco::Bugcheck::nullPointer(char const*, char const*, int) @ 0x2ebc5851 in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
    6. /src/ch/clickhouse/base/poco/Util/include/Poco/Util/Application.h:446: Poco::Util::Application::instance() @ 0x1ee3e6cd in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
    7. /src/ch/clickhouse/src/Interpreters/Context.cpp:358: DB::ContextSharedPart::ContextSharedPart() @ 0x263b0cca in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
    8. /src/ch/clickhouse/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714: std::__unique_if<DB::ContextSharedPart>::__unique_single std::make_unique[abi:v15000]<DB::ContextSharedPart>() @ 0x26387b6e in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
    9. /src/ch/clickhouse/src/Interpreters/Context.cpp:652: DB::Context::createShared() @ 0x26361a3c in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
    10. /src/ch/clickhouse/programs/obfuscator/Obfuscator.cpp:1293: mainEntryClickHouseObfuscator(int, char**) @ 0x1ef6eb26 in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
    11. /src/ch/clickhouse/programs/main.cpp:481: main @ 0x155e569a in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
    12. ? @ 0x7ffff7dd2790 in ?
    13. __libc_start_main @ 0x7ffff7dd284a in ?
    14. _start @ 0x155e51ae in /src/ch/clickhouse/.cmake-debug/programs/clickhouse
     (version 23.3.1.2537)

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-05 09:40:18 +02:00
Azat Khuzhin
a8c1407bc4 Use existing ServerSettings in Context instead of reading it again 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-05 09:40:18 +02:00
Azat Khuzhin
c332d290d8 Keep only one throttler for BACKUPs IO (instead of separate read/write) 
There is no need in separate read/write throttling, because you cannot
write faster then read anyway, and plus this makes the code less cleaner

(and also it will allow avoid implementing throttling backups to S3,
since it does not use common S3 writer).

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-05 09:40:17 +02:00
Azat Khuzhin
218b1f9c29 Add ability to throttle BACKUPs on per-server/backup basis 
Server settings:
- backup_read_bandwidth_for_server
- backup_write_bandwidth_for_server

Query settings:
- backup_read_bandwidth
- backup_write_bandwidth

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-05 09:39:48 +02:00
Azat Khuzhin
a25dd1d348 Add ability to throttle local IO on per-query/server basis 
Server settings:
- max_local_read_bandwidth_for_server
- max_local_write_bandwidth_for_server

Query settings:
- max_local_read_bandwidth
- max_local_write_bandwidth

This is the preparation for adding ability to throttle BACKUPs

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-05 09:39:15 +02:00
Azat Khuzhin
b3406beeb7 Add per-query network throttling 
Controlled with:
- max_remote_read_network_bandwidth
- max_remote_write_network_bandwidth

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-05 09:39:15 +02:00
Azat Khuzhin
abdb682048 Move some methods for Context into module part 
This will be required later, for per-query bandwidth limiting.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-05 09:39:15 +02:00
Azat Khuzhin
40cb87c2ac Make remote bandwidth settings server settings 
Right now those two settings are applied only at start:
- max_remote_read_network_bandwidth_for_server
- max_remote_write_network_bandwidth_for_server

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-04-05 09:39:15 +02:00
robot-clickhouse-ci-1
fb3af065f4
Merge pull request #48294 from ClickHouse/update-arrow-2 
Try to update arrow library to release 11.0.0
 2023-04-05 04:43:51 +02:00
FFFFFFFHHHHHHH
6e8f77ee9c
Add statistical aggregate function kolmogorovSmirnovTest (#48325) 2023-04-05 01:18:54 +02:00
Robert Schulze
db864891f8
Merge pull request #47610 from murfel/format-hiliting-fix 
Fix minor hiliting issues in clickhouse-format
 2023-04-04 22:59:46 +02:00
Dmitry Novik
96a1304730 WIP on additional_table_filters 2023-04-04 20:33:59 +00:00
Dmitry Novik
673413dc9f
Merge pull request #48355 from ClickHouse/lambda-type-resolution 
Fix lambda type resolution
 2023-04-04 22:07:57 +02:00
Igor Nikonov
0ff4e70c84 Fix 02579_fill_empty_chunk.sql 2023-04-04 19:11:08 +00:00
Alexander Tokmakov
ba65be22da fix bad error messages 2023-04-04 20:21:22 +02:00
vdimir
5966281f7c
Test WriteBufferToFileSegment with several threads 2023-04-04 18:15:27 +00:00
vdimir
61c8a9586e
Fix usage of is_unbound in FileSegment 2023-04-04 18:14:54 +00:00
Alexander Tokmakov
f5574d0d74 find big allocations without limits checks 2023-04-04 20:10:22 +02:00
Smita Kulkarni
a36b81a22d Fixed comment 2023-04-04 19:07:31 +02:00
Smita Kulkarni
f4e2d45fbc Added check for backup/restore when they fail and status is not COMPLETED 2023-04-04 19:05:27 +02:00
Igor Nikonov
078ebff92a Disable debug logging 2023-04-04 17:02:53 +00:00
Raúl Marín
3cee537e73 Changes for master 2023-04-04 18:58:28 +02:00
Raúl Marín
0e17b3b147 Merge remote-tracking branch 'blessed/master' into zk_retry_timeout 2023-04-04 18:53:55 +02:00
Smita Kulkarni
beb164dd51 Merge branch 'master' into Backup_Restore_concurrency_check_node_2 2023-04-04 18:51:30 +02:00
Igor Nikonov
2bf2764898 Fix typo 2023-04-04 16:43:48 +00:00
Igor Nikonov
1fb0292fcb polishing 2023-04-04 16:13:21 +00:00
Nikita Mikhaylov
fa5b2bd4a5
Added Keeper retries for backups operations (#47224) 2023-04-04 18:12:08 +02:00
Igor Nikonov
5a9acac12f Clarification and cleanup 2023-04-04 16:00:09 +00:00