thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-11 17:02:25 +00:00
Code Issues Packages Projects Releases Wiki Activity
133,091 Commits 9,299 Branches 1,782 Tags 2.1 GiB
90a0ea36b3
Commit Graph

12153 Commits

Author SHA1 Message Date
Azat Khuzhin
837f4ea676 Add ability to throttle merges/mutations 
Main motivation was to has an ability to throttle background tasks, to
avoid affecting queries.

To new server settings had been added for this:
- max_mutations_bandwidth_for_server
- max_merges_bandwidth_for_server

Note, that they limit only reading, since usually you will not write
more data then you read, but sometimes it is possible in case of ALTER
UPDATE.

But for now, to keep things simple, I decided to limit this with only
2 settings instead of 4.

Note, that if the write throttling will be needed, then they can use the
same settings, and just create new throttler for write.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-24 22:31:49 +01:00
Azat Khuzhin
3be3b0a280 Fix incorrect Exceptions 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-24 21:26:32 +01:00
Nikolay Degterinsky
033fb14a2a Merge remote-tracking branch 'upstream/master' into replicated-database-forbid-create-as-select 2023-12-24 20:07:12 +00:00
Alexey Milovidov
1054118607
Merge pull request #58133 from ClickHouse/parallel-replicas-in-subquery 
Disable parallel replicas with IN (subquery)
 2023-12-24 05:03:35 +01:00
Alexey Milovidov
3f4c8e4ae8
Merge pull request #58167 from jrdi/part-log-uncompressed-bytes 
Add bytes_uncompressed to system.part_log
 2023-12-24 04:11:35 +01:00
Igor Nikonov
3c85042679 Fix typo 2023-12-23 12:05:02 +00:00
Igor Nikonov
701c8ac9ba Cleanup & comments 2023-12-23 11:44:02 +00:00
Pradeep Chhetri
b5c8c4050b Add base backup name to system.backups and system.backup_log tables 2023-12-23 11:08:50 +08:00
jinjunzh
aaf98f0ed7 refine setting with enable_zstd_qat_codec 2023-12-22 14:24:15 -05:00
Jiebin Sun
d43448e747 Avoid the literal type from nullable 
Signed-off-by: Jiebin Sun <jiebin.sun@intel.com>
 2023-12-23 02:05:31 +08:00
Igor Nikonov
d3a0869119 Merge remote-tracking branch 'origin/master' into parallel-replicas-in-subquery 2023-12-22 17:54:45 +00:00
Igor Nikonov
ca1ddf0608 Fix style 2023-12-22 17:54:21 +00:00
Jordi Villar
b4c3969d3a Add bytes_uncompressed to system.part_log 2023-12-22 18:35:33 +01:00
Igor Nikonov
6f68696c14 Fix: for joins with old analyzer 2023-12-22 17:26:31 +00:00
vdimir
9b137059a3
Better logging in hash join 2023-12-22 15:52:47 +00:00
vdimir
4309d45730
Reserve memory in AddedColums 2023-12-22 15:52:47 +00:00
vdimir
47e46e0c50
Fix race in GraceHashJoin::DelayedBlocks 2023-12-22 15:52:46 +00:00
vdimir
6c9aa2ced5
handle not_processed in GraceHashJoin 2023-12-22 15:52:45 +00:00
vdimir
8429162038
HashJoin respects max_joined_block_size_rows 2023-12-22 15:52:41 +00:00
Nikolai Kochetov
3cbd895ce1
Merge pull request #57827 from ClickHouse/vdimir/merge_join_array_lowcard 
Fix low-cardinality keys support in MergeJoin
 2023-12-22 13:34:06 +01:00
Yarik Briukhovetskyi
483a377834
Merge pull request #57995 from yariks5s/paste_join 
Implementing PASTE JOIN
 2023-12-22 12:56:06 +01:00
Bharat Nallan Chakravarthy
62b28d365c Merge with upstream/master 2023-12-21 17:04:25 -08:00
Igor Nikonov
b9f1039ac4 Fix log message 2023-12-21 17:09:49 +00:00
Igor Nikonov
f0933d2714 Disable parallel replicas with IN with subquery 2023-12-21 16:21:02 +00:00
Raúl Marín
6d9da8edd5 Merge remote-tracking branch 'blessed/master' into zk_retries_quorum 2023-12-21 17:03:29 +01:00
Nikolay Degterinsky
7aa2363b40 Merge remote-tracking branch 'upstream/master' into replicated-database-forbid-create-as-select 2023-12-21 12:38:44 +00:00
Yakov Olkhovskiy
5dc88542cb
Merge branch 'master' into feature-server-iface-metrics 2023-12-20 16:46:29 -05:00
Igor Nikonov
db87ea0b86 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-20 20:48:51 +00:00
Nikita Mikhaylov
cb6c9557ff
Merge branch 'master' into remove-old-projections-code 2023-12-20 17:45:38 +01:00
Nikita Taranov
d57ac6dfdb
Set replica number to its position in cluster definition (#57800) 
* impl

* fix
 2023-12-20 17:33:05 +01:00
Alexander Tokmakov
015225c629
Merge pull request #57906 from ClickHouse/create_index_normalize 
Normalize function names in CREATE INDEX
 2023-12-20 17:24:19 +01:00
Alexander Tokmakov
858f2be6b5
Merge pull request #57337 from ClickHouse/drop_database_replica_no_throw 
Don't throw if noop when dropping database replica in batch
 2023-12-20 16:48:58 +01:00
Anton Popov
87d0cec055
Merge pull request #57433 from CurtizJ/materialize-lightweight-delete 
Add mutation command `APPLY DELETED MASK`
 2023-12-20 14:44:11 +01:00
Yakov Olkhovskiy
6830954cd3
Merge pull request #57988 from Chen768959/fix-literal-alias-misclassification 
Fix literal alias misclassification
 2023-12-19 19:36:59 -05:00
Alexey Milovidov
d190866fc0
Merge pull request #58031 from ClickHouse/revert-57766-fix-bug-window-functions 
Revert "Fix bug window functions: revert #39631"
 2023-12-20 01:08:08 +01:00
Raúl Marín
ccff198265
Rename canUseParallelReplicas to canUseTaskBasedParallelReplicas (#58025) 2023-12-19 23:21:19 +01:00
Jordi Villar
7c281d9a6f
Allow max_size_to_drop settings in query time (#57452) 2023-12-19 23:16:03 +01:00
Yarik Briukhovetskyi
6e816d45ae
Merge branch 'master' into paste_join 2023-12-19 20:02:24 +01:00
yariks5s
4e418a89d9 fixed suggestions 2023-12-19 19:00:37 +00:00
Sema Checherinda
802ad4ec47
Merge pull request #57523 from ClickHouse/remove-the-limit-for-connections-per-endpoint 
Remove the limit for the number of connections per endpoint for background fetches + small refactoring
 2023-12-19 19:16:01 +01:00
Bharat Nallan Chakravarthy
ecd2843b47 add a clarification to docs 2023-12-19 08:35:14 -08:00
Kseniia Sumarokova
d4e71f96c1
Merge pull request #57897 from ClickHouse/allow-to-dynamically-change-fs-cache-size 
Support dynamic reloading of filesystem cache size
 2023-12-19 13:51:42 +01:00
Robert Schulze
952175ce39
Merge pull request #57875 from zhongyuankai/order_by_all 
Support `ORDER BY ALL`
 2023-12-19 13:38:07 +01:00
vdimir
252cb8a507
Merge pull request #54602 from fionera/issue-49028 
Add setting http_make_head_request
 2023-12-19 13:33:59 +01:00
Nikolai Kochetov
8ab6564538
Revert "Fix bug window functions: revert #39631" 2023-12-19 13:30:59 +01:00
Chen768959
c1f7de770d
Merge branch 'ClickHouse:master' into fix-literal-alias-misclassification 2023-12-19 19:48:25 +08:00
Anton Popov
34cdecea70
Merge pull request #58013 from CurtizJ/fix-race-external-sort 
Fix rare race in external sort/aggregation with temporary data in cache
 2023-12-19 12:20:31 +01:00
Nikita Mikhaylov
6360b76792 Merge branch 'master' of github.com:ClickHouse/ClickHouse into remove-the-limit-for-connections-per-endpoint 2023-12-18 21:49:31 +00:00
pufit
36d1905881
Merge pull request #57752 from chizhonkova/refactor_sql_udf_loader 
Refactoring for UserDefinedSQLObjectsLoader
 2023-12-18 16:08:46 -05:00
Anton Popov
27d632f5e4 fix race in operations with external temporary data in cache 2023-12-18 20:02:53 +00:00
Alexey Milovidov
7178bd5c88
Merge pull request #57910 from Algunenano/log_message_string 
Improvements for 00002_log_and_exception_messages_formatting
 2023-12-18 21:02:23 +01:00
Bharat Nallan Chakravarthy
aef4e2e6ec Merge upstream/master into ncb/async-metrics-pri-key-size 2023-12-18 09:46:51 -08:00
Raúl Marín
46067adce8 Improve ZooKeeperRetriesControl so it's easier to reuse 2023-12-18 17:43:10 +01:00
Nikolai Kochetov
5290b3c9ce
Merge pull request #57855 from kitaisreal/interpreter-create-query-sample-block-fix 
InterpreterCreateQuery sample block fix
 2023-12-18 17:22:24 +01:00
yariks5s
6740316a88 init 2023-12-18 15:02:51 +00:00
Yakov Olkhovskiy
f99b36143f Merge remote-tracking branch 'origin/master' into feature-server-iface-metrics 2023-12-18 14:14:03 +00:00
Anton Popov
10bc2db56d add setting 2023-12-18 13:57:07 +00:00
chen768959
fdda0cf8ac Literal aliases incorrectly classified as 'complex_aliases'. 2023-12-18 21:43:07 +08:00
Kseniia Sumarokova
f11fc650ee
Merge branch 'master' into allow-to-dynamically-change-fs-cache-size 2023-12-18 13:10:36 +01:00
Igor Nikonov
52db759c92 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-18 11:59:13 +00:00
Raúl Marín
2639d0715f Merge remote-tracking branch 'blessed/master' into log_message_string 2023-12-18 10:40:18 +01:00
Raúl Marín
f10dae4761
Merge pull request #56917 from Algunenano/backup_1 
Prepare the introduction of more keeper faults
 2023-12-18 10:27:48 +01:00
Alexey Milovidov
b07ee86cec
Merge branch 'master' into allow-to-dynamically-change-fs-cache-size 2023-12-18 05:50:31 +01:00
Alexey Milovidov
52d7777b05 Merge branch 'remove-cpp-templates-3' of github.com:ClickHouse/ClickHouse into remove-cpp-templates-3 2023-12-18 04:53:06 +01:00
Alexey Milovidov
c05b0cf37f Merge branch 'master' into remove-cpp-templates-3 2023-12-18 04:52:50 +01:00
Alexey Milovidov
58396c5546
Merge pull request #57218 from tntnatbry/issue-43666 
Issue 43666: Add skip_unavailable_shards as a setting for Distributed table.
 2023-12-18 04:48:57 +01:00
Alexey Milovidov
168da20d51
Update executeQuery.cpp 2023-12-17 22:54:45 +03:00
Igor Nikonov
af6401761e Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-17 13:32:53 +00:00
Robert Schulze
9033c96e29
Some fixups 2023-12-17 09:11:16 +00:00
Igor Nikonov
c49de7cea4 Fix typo 2023-12-17 08:33:49 +00:00
ubuntu
a5e4495ee5 Merge branch 'master' into order_by_all 2023-12-17 10:07:35 +08:00
Alexey Milovidov
6ada95ea00 Remove C++ templates (normalizeQuery) 2023-12-17 02:44:30 +01:00
Igor Nikonov
ec1539728d Fix load balancing for hedged connections 2023-12-16 21:17:01 +00:00
Nikolay Degterinsky
5a3f0e7a5b Forbid CREATE AS SELECT for Replicated storages only 2023-12-16 14:25:27 +00:00
zhongyuankai
5ccc5b4fd8 batter 2023-12-16 18:49:25 +08:00
zhongyuankai
01f5b20517 Support ORDER BY ALL 2023-12-16 18:48:23 +08:00
Kseniia Sumarokova
4279dd2bf1
Merge pull request #57687 from ClickHouse/make-system-filesystem-cache-query-no-memory-intensive 
Make querying system.filesystem_cache not memory intensive
 2023-12-16 11:28:28 +01:00
Alexey Milovidov
2d58dc512c
Merge pull request #57932 from ClickHouse/remove-shit-cleanup 
Remove a feature
 2023-12-16 06:04:59 +01:00
Anton Popov
5faf5e913e slightly faster and perf test 2023-12-16 03:35:59 +00:00
jsc0218
41873dc4a3
Merge pull request #57873 from ClickHouse/vdimir/hash_join_structure_equals_fix 
Fix assertion in HashJoin
 2023-12-15 21:35:08 -05:00
Alexey Milovidov
ee502ef6b9 Remove trash (OPTIMIZE CLEANUP) 2023-12-15 22:41:58 +01:00
Igor Nikonov
a735820400 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-15 21:14:43 +00:00
Alexey Milovidov
d751561d44 Remove useless headers 2023-12-15 21:42:05 +01:00
Alexey Milovidov
7e55eab1c3 Revert "Merge pull request #56573 from mkmkme/mkmkme/reload-config" 
This reverts commit 3067ca64df, reversing
changes made to 1e46460910.
 2023-12-15 19:28:25 +01:00
Raúl Marín
b269f87f4c Better text_log with ErrnoException 2023-12-15 19:27:56 +01:00
Anton Popov
a226c75717 optimize usage of consecutive keys optimization 2023-12-15 16:58:45 +00:00
Alexander Tokmakov
e4a831be17 normalize function names in CREATE INDEX 2023-12-15 17:40:03 +01:00
Maksim Kita
a4d287b576 Fixed tests 2023-12-15 18:54:15 +03:00
kssenii
0dcb3133c7 Allow to dynamically change fs cache size 2023-12-15 14:11:52 +01:00
Yarik Briukhovetskyi
325374c68b
Merge pull request #57342 from bharatnc/ncb/better-hint-if-table-doesnt-exist 
provide a better hint if a table doesn't exist
 2023-12-15 12:35:13 +01:00
vdimir
16c7c1e308
fix 2023-12-15 10:58:33 +00:00
kssenii
5de9c143a5 Merge remote-tracking branch 'origin/master' into make-system-filesystem-cache-query-no-memory-intensive 2023-12-15 11:32:14 +01:00
Kseniia Sumarokova
06438cc17b
Merge pull request #57578 from ClickHouse/allow-to-change-some-cache-settings-without-restart 
Allow to apply some fs cache config settings changes without server restart
 2023-12-15 11:18:19 +01:00
Maksim Kita
8a5ffbb5f2 InterpreterCreateQuery sample block fix 2023-12-15 12:55:19 +03:00
Bharat Nallan Chakravarthy
c9f35a8f84 review fixes 2023-12-14 11:28:49 -08:00
Bharat Nallan Chakravarthy
daef2febf3 Merge with 'upstream/master' 2023-12-14 11:19:44 -08:00
Yakov Olkhovskiy
febdd542f8 Merge remote-tracking branch 'origin/master' into feature-server-iface-metrics 2023-12-14 18:59:44 +00:00
Igor Nikonov
1370a8c6c4 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-14 18:29:38 +00:00
Raúl Marín
546484d46b Merge remote-tracking branch 'blessed/master' into backup_1 2023-12-14 19:03:59 +01:00
Jiebin Sun
b3f3a0c75c Rewrite the AST of sum(column + literal) function 
Rewrite the following AST of `sum(column + literal)`
sum(column + literal)  ->  sum(column) + literal * count(column)
sum(literal + column)  ->  sum(column) + literal * count(column)

Test the patch on 2 x 80 vCPUs system, Q29 of ClickBench has got a huge
11.5x performance improvement.

Signed-off-by: Jiebin Sun <jiebin.sun@intel.com>
 2023-12-15 01:28:47 +08:00
Yakov Olkhovskiy
6d20082134 return the early release and instead restore prepared_client_info and reset session_context 2023-12-14 17:25:47 +00:00
vdimir
eb350ffa7e
Fix assertion in HashJoin 2023-12-14 15:52:35 +00:00
Anton Popov
5630115faa optimize aggregation by large ranges of equal keys 2023-12-14 15:38:01 +00:00
kssenii
12ed9952c4 Merge remote-tracking branch 'origin/allow-to-change-some-cache-settings-without-restart' into allow-to-change-some-cache-settings-without-restart 2023-12-14 16:31:13 +01:00
kssenii
3edbbab9a3 Better 2023-12-14 16:30:15 +01:00
vdimir
0473e39352
lowcard in join output 2023-12-14 15:28:19 +00:00
kssenii
ea7e55b929 Simplify 2023-12-14 15:58:06 +01:00
Nikita Mikhaylov
a0af0392cd
Random changes in random files (#57642) 2023-12-14 12:47:11 +01:00
János Benjamin Antal
fdfd1481df
Merge pull request #57610 from kitaisreal/jit-add-assembly-printer 
CHJIT add assembly printer
 2023-12-14 12:39:20 +01:00
Raúl Marín
efcacd3c34 Merge remote-tracking branch 'blessed/master' into backup_1 2023-12-14 11:08:50 +01:00
Kseniia Sumarokova
79db3c66df
Merge branch 'master' into allow-to-change-some-cache-settings-without-restart 2023-12-13 23:33:59 +01:00
kssenii
38e38ef43f Minor fix 2023-12-13 19:32:47 +01:00
Nikita Mikhaylov
8372c70958 Merge branch 'master' of github.com:ClickHouse/ClickHouse into remove-the-limit-for-connections-per-endpoint 2023-12-13 18:29:56 +00:00
vdimir
48fe75a70d
Fix MergeJoin lowcard keys support 2023-12-13 17:43:15 +00:00
Anton Popov
1960713176 add docs 2023-12-13 17:29:34 +00:00
Anton Popov
7598403f78 Merge remote-tracking branch 'upstream/master' into HEAD 2023-12-13 16:55:17 +00:00
Raúl Marín
d4522ee088
Merge pull request #57811 from Algunenano/revert_negative_positional_arguments 
Revert #57741
 2023-12-13 16:39:23 +01:00
Igor Nikonov
72f1c415f1 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-13 15:03:05 +00:00
Igor Nikonov
c165be76ab
Parallel replicas: friendly settings (#57542) 2023-12-13 14:42:06 +01:00
Raúl Marín
dd405a655e Merge remote-tracking branch 'blessed/master' into backup_1 2023-12-13 13:39:58 +01:00
Raúl Marín
8c2137e0c6 Revert "Merge pull request #57741 from ucasfl/negtive-position" 
This reverts commit 3d846800e0, reversing
changes made to b31b4c932f.
 2023-12-13 13:09:36 +01:00
Nikolay Degterinsky
254bcade1e Forbid CREATE AS SELECT for database Replicated 2023-12-12 18:28:20 +00:00
kssenii
cc71b40002 Merge remote-tracking branch 'origin/master' into make-system-filesystem-cache-query-no-memory-intensive 2023-12-12 16:40:04 +01:00
kssenii
fa0b2b9e50 Merge remote-tracking branch 'origin/master' into make-system-filesystem-cache-query-no-memory-intensive 2023-12-12 15:31:01 +01:00
Han Fei
a96593c8b1 Merge branch 'master' into feature/more_warnings 2023-12-12 15:25:01 +01:00
Natalya Chizhonkova
3d51192bad Refactoring for UserDefinedSQLObjectsLoader 2023-12-12 17:03:49 +03:00
Kruglov Pavel
b33f124555
Merge pull request #57538 from MikhailBurdukov/ignore_on_cluster_for_grants 
Ignore ON CLUSTER clause in grant/revoke queries for management of replicated access entities.
 2023-12-12 14:59:58 +01:00
Nikolai Kochetov
696a2d9f4f
Merge pull request #57614 from ClickHouse/vdimir/join_array_lowcard_totals_crash 
Fix type correction in HashJoin for nested low cardinality
 2023-12-12 14:56:41 +01:00
Kruglov Pavel
3d846800e0
Merge pull request #57741 from ucasfl/negtive-position 
Support negative positional arguments
 2023-12-12 14:16:10 +01:00
Kseniia Sumarokova
91d36ad224
Merge pull request #57076 from ClickHouse/slru-for-filesystem-cache 
Implement SLRU cache policy for filesystem cache
 2023-12-12 10:20:58 +01:00
flynn
02720cde01 Fix 2023-12-12 04:06:43 +00:00
Alexey Milovidov
22200566c8 Revert "Merge pull request #39631 from ClickHouse/parallel-window" 
This reverts commit 33cc853d61, reversing
changes made to b05a3d02ed.
 2023-12-12 03:45:00 +01:00
kssenii
554ef59dcf Better 2023-12-11 21:34:07 +01:00
vdimir
eaf2c34761
better lowcard handling in hash join 2023-12-11 18:35:38 +00:00
vdimir
ea09080c73
fix 2023-12-11 15:50:27 +00:00
Raúl Marín
a55a0c068e Merge remote-tracking branch 'blessed/master' into backup_1 2023-12-11 15:27:21 +01:00
kssenii
e10d06af6f Better 2023-12-11 14:07:28 +01:00
vdimir
af45e138ad
fix 2023-12-11 12:33:11 +00:00
vdimir
cdfe99c380
Remove unused template parameter from fillColumnsFromMap 2023-12-11 11:34:54 +00:00
kssenii
d34a2782ff Allow to decrease the number of download threads 2023-12-11 12:17:26 +01:00
flynn
c78649bdcf Support negative positional arguments 2023-12-11 10:36:26 +00:00
kssenii
3652419873 Update run.sh 2023-12-11 11:01:13 +01:00
Alexey Milovidov
27f19635cd Increase load_metadata_threads to 16 2023-12-11 00:49:20 +01:00
Igor Nikonov
f7e6a17052 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-10 20:21:47 +00:00
Azat Khuzhin
450e8da7ce Fix data race on vptr (ctor/dtor vs virtual call) in AsynchronousMetrics 
TSan report [1]:

     Exception: Sanitizer assert found for instance ==================
     WARNING: ThreadSanitizer: data race on vptr (ctor/dtor vs virtual call) (pid=1)
       Write of size 8 at 0x7ffce68f5680 by main thread:
         0 DB::AsynchronousMetrics::~AsynchronousMetrics() build_docker/./src/Common/AsynchronousMetrics.cpp:299:1 (clickhouse+0xf38fa6e) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         1 DB::ServerAsynchronousMetrics::~ServerAsynchronousMetrics() build_docker/./src/Interpreters/ServerAsynchronousMetrics.h:10:7 (clickhouse+0xf2b2f0f) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         2 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&) build_docker/./programs/server/Server.cpp:1996:1 (clickhouse+0xf2b2f0f)
         3 Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 (clickhouse+0x1d6c95fe) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         4 DB::Server::run() build_docker/./programs/server/Server.cpp:397:25 (clickhouse+0xf29e041) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         5 Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 (clickhouse+0x1d6e9354) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         6 mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:203:20 (clickhouse+0xf29b1a3) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         7 main build_docker/./programs/main.cpp:505:12 (clickhouse+0x72d7fa0) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)

       Previous read of size 8 at 0x7ffce68f5680 by thread T649:
         0 DB::AsynchronousMetrics::update(std::__1::chrono::time_point<std::__1::chrono::system_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000l>>>) build_docker/./src/Common/AsynchronousMetrics.cpp:1559:5 (clickhouse+0xf38ebc5) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         1 DB::AsynchronousMetrics::run() build_docker/./src/Common/AsynchronousMetrics.cpp:354:13 (clickhouse+0xf390696) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         2 DB::AsynchronousMetrics::start()::$_0::operator()() const build_docker/./src/Common/AsynchronousMetrics.cpp:273:62 (clickhouse+0xf392eca) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         3 decltype(std::declval<DB::AsynchronousMetrics::start()::$_0&>()()) std::__1::__invoke[abi:v15000]<DB::AsynchronousMetrics::start()::$_0&>(DB::AsynchronousMetrics::start()::$_0&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xf392eca)
         4 decltype(auto) std::__1::__apply_tuple_impl[abi:v15000]<DB::AsynchronousMetrics::start()::$_0&, std::__1::tuple<>&>(DB::AsynchronousMetrics::start()::$_0&, std::__1::tuple<>&, std::__1::__tuple_indices<>) build_docker/./contrib/llvm-project/libcxx/include/tuple:1789:1 (clickhouse+0xf392eca)
         5 decltype(auto) std::__1::apply[abi:v15000]<DB::AsynchronousMetrics::start()::$_0&, std::__1::tuple<>&>(DB::AsynchronousMetrics::start()::$_0&, std::__1::tuple<>&) build_docker/./contrib/llvm-project/libcxx/include/tuple:1798:1 (clickhouse+0xf392eca)
         6 ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'()::operator()() build_docker/./src/Common/ThreadPool.h:253:13 (clickhouse+0xf392eca)
         7 decltype(std::declval<DB::AsynchronousMetrics::start()::$_0>()()) std::__1::__invoke[abi:v15000]<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'()&>(DB::AsynchronousMetrics::start()::$_0&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xf392eca)
         8 void std::__1::__invoke_void_return_wrapper<void, true>::__call<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'()&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:479:9 (clickhouse+0xf392eca)
         9 std::__1::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12 (clickhouse+0xf392eca)
         10 void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'(), void ()>>(std::__1::__function::__policy_storage const*) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16 (clickhouse+0xf392eca)
         11 std::__1::__function::__policy_func<void ()>::operator()[abi:v15000]() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16 (clickhouse+0xf30682e) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         12 std::__1::function<void ()>::operator()() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:1187:12 (clickhouse+0xf30682e)
         13 ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) build_docker/./src/Common/ThreadPool.cpp:421:13 (clickhouse+0xf30682e)
         14 void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()::operator()() const build_docker/./src/Common/ThreadPool.cpp:183:73 (clickhouse+0xf30d3d1) (BuildId: d49dc55f80b62802f8721fdbcb07bdc88ac204a4)
         15 decltype(std::declval<void>()()) std::__1::__invoke[abi:v15000]<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xf30d3d1)
         16 void std::__1::__thread_execute[abi:v15000]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(std::__1::tuple<void, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>&, std::__1::__tuple_indices<>) build_docker/./contrib/llvm-project/libcxx/include/thread:284:5 (clickhouse+0xf30d3d1)
         17 void* std::__1::__thread_proxy[abi:v15000]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>>(void*) build_docker/./contrib/llvm-project/libcxx/include/thread:295:5 (clickhouse+0xf30d3d1)

     SUMMARY: ThreadSanitizer: data race on vptr (ctor/dtor vs virtual call) build_docker/./src/Common/AsynchronousMetrics.cpp:299:1 in DB::AsynchronousMetrics::~AsynchronousMetrics()

And the problem is real, see [2].

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/52717/a6cf2297ef07ca0b065d944c9e30a3986285331d/integration_tests__tsan__[2_6]/integration_run_parallel0_0.log
  [2]: https://github.com/google/sanitizers/wiki/ThreadSanitizerPopularDataRaces#data-race-on-vptr

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-10 18:32:40 +01:00
Azat Khuzhin
a6cf2297ef Fix Context use-after-free in ServerAsynchronousMetrics 
TSan found [1]:

    WARNING: ThreadSanitizer: data race on vptr (ctor/dtor vs virtual call) (pid=598)
      Write of size 8 at 0x7ffcf9ad9cb0 by main thread:
        0 DB::AsynchronousMetrics::~AsynchronousMetrics() build_docker/./src/Common/AsynchronousMetrics.cpp:299:1 (clickhouse+0xf38ed6e) (BuildId: 31dcc7d77a0b2aaf9de7aca070b5f6ed6ac3dcbf)
        1 DB::ServerAsynchronousMetrics::~ServerAsynchronousMetrics() build_docker/./src/Interpreters/ServerAsynchronousMetrics.h:10:7 (clickhouse+0xf2b2220) (BuildId: 31dcc7d77a0b2aaf9de7aca070b5f6ed6ac3dcbf)
        2 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&) build_docker/./programs/server/Server.cpp:1995:1 (clickhouse+0xf2b2220)
        3 Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8 (clickhouse+0x1d6c535e) (BuildId: 31dcc7d77a0b2aaf9de7aca070b5f6ed6ac3dcbf)
        4 DB::Server::run() build_docker/./programs/server/Server.cpp:397:25 (clickhouse+0xf29d341) (BuildId: 31dcc7d77a0b2aaf9de7aca070b5f6ed6ac3dcbf)
        5 Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9 (clickhouse+0x1d6e50b4) (BuildId: 31dcc7d77a0b2aaf9de7aca070b5f6ed6ac3dcbf)
        6 mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:203:20 (clickhouse+0xf29a4a3) (BuildId: 31dcc7d77a0b2aaf9de7aca070b5f6ed6ac3dcbf)
        7 main build_docker/./programs/main.cpp:505:12 (clickhouse+0x72d72a0) (BuildId: 31dcc7d77a0b2aaf9de7aca070b5f6ed6ac3dcbf)

      Previous read of size 8 at 0x7ffcf9ad9cb0 by thread T727:
        0 DB::AsynchronousMetrics::update(std::__1::chrono::time_point<std::__1::chrono::system_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000l>>>) build_docker/./src/Common/AsynchronousMetrics.cpp:1559:5 (clickhouse+0xf38dec5) (BuildId: 31dcc7d77a0b2aaf9de7aca070b5f6ed6ac3dcbf)
        1 DB::AsynchronousMetrics::run() build_docker/./src/Common/AsynchronousMetrics.cpp:354:13 (clickhouse+0xf38f996) (BuildId: 31dcc7d77a0b2aaf9de7aca070b5f6ed6ac3dcbf)
        2 DB::AsynchronousMetrics::start()::$_0::operator()() const build_docker/./src/Common/AsynchronousMetrics.cpp:273:62 (clickhouse+0xf3921ca) (BuildId: 31dcc7d77a0b2aaf9de7aca070b5f6ed6ac3dcbf)
        3 decltype(std::declval<DB::AsynchronousMetrics::start()::$_0&>()()) std::__1::__invoke[abi:v15000]<DB::AsynchronousMetrics::start()::$_0&>(DB::AsynchronousMetrics::start()::$_0&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xf3921ca)
        4 decltype(auto) std::__1::__apply_tuple_impl[abi:v15000]<DB::AsynchronousMetrics::start()::$_0&, std::__1::tuple<>&>(DB::AsynchronousMetrics::start()::$_0&, std::__1::tuple<>&, std::__1::__tuple_indices<>) build_docker/./contrib/llvm-project/libcxx/include/tuple:1789:1 (clickhouse+0xf3921ca)
        5 decltype(auto) std::__1::apply[abi:v15000]<DB::AsynchronousMetrics::start()::$_0&, std::__1::tuple<>&>(DB::AsynchronousMetrics::start()::$_0&, std::__1::tuple<>&) build_docker/./contrib/llvm-project/libcxx/include/tuple:1798:1 (clickhouse+0xf3921ca)
        6 ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'()::operator()() build_docker/./src/Common/ThreadPool.h:253:13 (clickhouse+0xf3921ca)
        7 decltype(std::declval<DB::AsynchronousMetrics::start()::$_0>()()) std::__1::__invoke[abi:v15000]<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'()&>(DB::AsynchronousMetrics::start()::$_0&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xf3921ca)
        8 void std::__1::__invoke_void_return_wrapper<void, true>::__call<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'()&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:479:9 (clickhouse+0xf3921ca)
        9 std::__1::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12 (clickhouse+0xf3921ca)
        10 void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'(), void ()>>(std::__1::__function::__policy_storage const*) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16 (clickhouse+0xf3921ca)
        11 std::__1::__function::__policy_func<void ()>::operator()[abi:v15000]() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16 (clickhouse+0xf305b2e) (BuildId: 31dcc7d77a0b2aaf9de7aca070b5f6ed6ac3dcbf)
        12 std::__1::function<void ()>::operator()() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:1187:12 (clickhouse+0xf305b2e)
        13 ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) build_docker/./src/Common/ThreadPool.cpp:421:13 (clickhouse+0xf305b2e)
        14 void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()::operator()() const build_docker/./src/Common/ThreadPool.cpp:183:73 (clickhouse+0xf30c6d1) (BuildId: 31dcc7d77a0b2aaf9de7aca070b5f6ed6ac3dcbf)
        15 decltype(std::declval<void>()()) std::__1::__invoke[abi:v15000]<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xf30c6d1)
        16 void std::__1::__thread_execute[abi:v15000]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(std::__1::tuple<void, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>&, std::__1::__tuple_indices<>) build_docker/./contrib/llvm-project/libcxx/include/thread:284:5 (clickhouse+0xf30c6d1)
        17 void* std::__1::__thread_proxy[abi:v15000]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>>(void*) build_docker/./contrib/llvm-project/libcxx/include/thread:295:5 (clickhouse+0xf30c6d1)

    SUMMARY: ThreadSanitizer: data race on vptr (ctor/dtor vs virtual call) build_docker/./src/Common/AsynchronousMetrics.cpp:299:1 in DB::AsynchronousMetrics::~AsynchronousMetrics()

Or MSan [2]:

    ==573==WARNING: MemorySanitizer: use-of-uninitialized-value
        0 0x55b57079112d in std::__1::weak_ptr<DB::Context const>::lock() const build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:1645:20
        1 0x55b57079112d in DB::WithContextImpl<std::__1::shared_ptr<DB::Context const>>::getContext() const build_docker/./src/Interpreters/Context_fwd.h:41:28
        2 0x55b57079112d in DB::ServerAsynchronousMetrics::updateImpl(std::__1::unordered_map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, DB::AsynchronousMetricValue, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, DB::AsynchronousMetricValue>>>&, std::__1::chrono::time_point<std::__1::chrono::system_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000l>>>, std::__1::chrono::time_point<std::__1::chrono::system_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000l>>>) build_docker/./src/Interpreters/ServerAsynchronousMetrics.cpp:260:63
        3 0x55b559540309 in DB::AsynchronousMetrics::update(std::__1::chrono::time_point<std::__1::chrono::system_clock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000l>>>) build_docker/./src/Common/AsynchronousMetrics.cpp:1559:5
        4 0x55b55954258c in DB::AsynchronousMetrics::run() build_docker/./src/Common/AsynchronousMetrics.cpp:354:13
        5 0x55b559549111 in DB::AsynchronousMetrics::start()::$_0::operator()() const build_docker/./src/Common/AsynchronousMetrics.cpp:273:62
        6 0x55b559549111 in decltype(std::declval<DB::AsynchronousMetrics::start()::$_0&>()()) std::__1::__invoke[abi:v15000]<DB::AsynchronousMetrics::start()::$_0&>(DB::AsynchronousMetrics::start()::$_0&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23
        7 0x55b559549111 in decltype(auto) std::__1::__apply_tuple_impl[abi:v15000]<DB::AsynchronousMetrics::start()::$_0&, std::__1::tuple<>&>(DB::AsynchronousMetrics::start()::$_0&, std::__1::tuple<>&, std::__1::__tuple_indices<>) build_docker/./contrib/llvm-project/libcxx/include/tuple:1789:1
        8 0x55b559549111 in decltype(auto) std::__1::apply[abi:v15000]<DB::AsynchronousMetrics::start()::$_0&, std::__1::tuple<>&>(DB::AsynchronousMetrics::start()::$_0&, std::__1::tuple<>&) build_docker/./contrib/llvm-project/libcxx/include/tuple:1798:1
        9 0x55b559549111 in ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'()::operator()() build_docker/./src/Common/ThreadPool.h:253:13
        10 0x55b559549111 in decltype(std::declval<DB::AsynchronousMetrics::start()::$_0>()()) std::__1::__invoke[abi:v15000]<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'()&>(DB::AsynchronousMetrics::start()::$_0&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23
        11 0x55b559549111 in void std::__1::__invoke_void_return_wrapper<void, true>::__call<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'()&>(ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'()&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:479:9
        12 0x55b559549111 in std::__1::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'(), void ()>::operator()[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12
        13 0x55b559549111 in void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::AsynchronousMetrics::start()::$_0>(DB::AsynchronousMetrics::start()::$_0&&)::'lambda'(), void ()>>(std::__1::__function::__policy_storage const*) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16
        14 0x55b5593eb38a in std::__1::__function::__policy_func<void ()>::operator()[abi:v15000]() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16
        15 0x55b5593eb38a in std::__1::function<void ()>::operator()() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:1187:12
        16 0x55b5593eb38a in ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) build_docker/./src/Common/ThreadPool.cpp:421:13
        17 0x55b5593f9a0a in void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()::operator()() const build_docker/./src/Common/ThreadPool.cpp:183:73
        18 0x55b5593f9a0a in decltype(std::declval<void>()()) std::__1::__invoke[abi:v15000]<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23
        19 0x55b5593f9a0a in void std::__1::__thread_execute[abi:v15000]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>(std::__1::tuple<void, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>&, std::__1::__tuple_indices<>) build_docker/./contrib/llvm-project/libcxx/include/thread:284:5
        20 0x55b5593f9a0a in void* std::__1::__thread_proxy[abi:v15000]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, Priority, std::__1::optional<unsigned long>, bool)::'lambda0'()>>(void*) build_docker/./contrib/llvm-project/libcxx/include/thread:295:5
        21 0x7f7ff3899ac2  (/lib/x86_64-linux-gnu/libc.so.6+0x94ac2) (BuildId: a43bfc8428df6623cd498c9c0caeb91aec9be4f9)
        22 0x7f7ff392ba3f  (/lib/x86_64-linux-gnu/libc.so.6+0x126a3f) (BuildId: a43bfc8428df6623cd498c9c0caeb91aec9be4f9)

      Member fields were destroyed
        0 0x55b541a72c9d in __sanitizer_dtor_callback_fields (/usr/bin/clickhouse+0x7c6dc9d) (BuildId: 57941f7730deefd0a2028f1d9f3e173472a4aa76)
        1 0x55b559304b83 in std::__1::weak_ptr<DB::Context const>::~weak_ptr() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:1397:26
        2 0x55b559304b83 in std::__1::weak_ptr<DB::Context const>::~weak_ptr() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:1553:1
        3 0x55b559304b83 in DB::WithContextImpl<std::__1::shared_ptr<DB::Context const>>::~WithContextImpl() build_docker/./src/Interpreters/Context_fwd.h:30:8
        4 0x55b559304b83 in DB::ServerAsynchronousMetrics::~ServerAsynchronousMetrics() build_docker/./src/Interpreters/ServerAsynchronousMetrics.h:10:7
        5 0x55b559304b83 in DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&) build_docker/./programs/server/Server.cpp:1995:1
        6 0x55b5789f579e in Poco::Util::Application::run() build_docker/./base/poco/Util/src/Application.cpp:315:8
        7 0x55b5592d762a in DB::Server::run() build_docker/./programs/server/Server.cpp:397:25
        8 0x55b578a3efdf in Poco::Util::ServerApplication::run(int, char**) build_docker/./base/poco/Util/src/ServerApplication.cpp:131:9
        9 0x55b5592d0489 in mainEntryClickHouseServer(int, char**) build_docker/./programs/server/Server.cpp:203:20
        10 0x55b541acd4ab in main build_docker/./programs/main.cpp:505:12
        11 0x7f7ff382ed8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: a43bfc8428df6623cd498c9c0caeb91aec9be4f9)

SUMMARY: MemorySanitizer: use-of-uninitialized-value build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:1645:20 in std::__1::weak_ptr<DB::Context const>::lock() const

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/52717/fcdead023c4350233ef1e0f7f82a71653ed62229/stress_test__tsan_.html
  [2]: https://s3.amazonaws.com/clickhouse-test-reports/52717/fcdead023c4350233ef1e0f7f82a71653ed62229/stress_test__msan_.html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-10 16:03:52 +01:00
Maksim Kita
cdf6da88fe Fixed code review issues 2023-12-10 16:10:11 +03:00
Maksim Kita
0eb7a41bab CHJIT add assembly printer 2023-12-10 16:10:11 +03:00
Azat Khuzhin
6ccbc2ea75 Move io_uring reader into the Context from static to make it's thread joinable 
v2: fix for standalone keeper build
CI: https://s3.amazonaws.com/clickhouse-test-reports/52717/72b1052f7c2d453308262924e767ab8dc2206933/stateless_tests__debug__[4_5].html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-09 22:50:48 +01:00
Nikolay Degterinsky
c5e52d260f
Merge pull request #57553 from evillique/fix-local-crash 
Fix crash in clickhouse-local
 2023-12-08 22:57:52 +01:00
Gagan Goel
e547db0a8c Issue 43666: Add skip_unavailable_shards as a setting for Distributed table. 
This setting, when enabled (disabled by default), allows ClickHouse to
silently skip unavailable shards of a Distributed table during a query
execution, instead of throwing an exception to the client.
 2023-12-08 15:43:59 -05:00
Nikolay Degterinsky
10bfd054d8 Fix style & review 2023-12-08 19:40:09 +00:00
Nikita Mikhaylov
c979124cfe Merge branch 'master' of github.com:ClickHouse/ClickHouse into remove-the-limit-for-connections-per-endpoint 2023-12-08 16:25:02 +00:00
kssenii
02d19fa7ef Make querying system.filesystem_cache not memory intensive 2023-12-08 17:09:09 +01:00
vdimir
8ebd5a2d6c
Fix type correction in HashJoin for nested low cardinality 2023-12-08 15:50:33 +00:00
Kseniia Sumarokova
bd3665863c
Merge branch 'master' into allow-to-change-some-cache-settings-without-restart 2023-12-08 15:36:40 +01:00
kssenii
ea63819c76 Better 2023-12-08 15:21:42 +01:00
zhongyuankai
7b0f8d44e8
Make DirectoryMonitor handle cluster node list change (#42826) 2023-12-08 14:41:51 +01:00
Alexey Milovidov
429ed34607
Merge pull request #57479 from mkmkme/mkmkme/zookeeper-read-only 
Add support for read-only mode in ZooKeeper
 2023-12-07 20:22:00 +01:00
kssenii
ff65d0e727 Adjustments after merge with master 2023-12-07 19:49:30 +01:00
kssenii
725571461d Merge remote-tracking branch 'origin/master' into slru-for-filesystem-cache 2023-12-07 19:49:03 +01:00
kssenii
7205e4edbe Review fix 2023-12-07 18:47:14 +01:00
vdimir
f058fe1d20
Merge pull request #57564 from ClickHouse/vdimir/join_on_false_const_fix 
Materialize block in HashJoin for Type::EMPTY
 2023-12-07 15:09:05 +01:00
kssenii
c785ebe921 Fix style check 2023-12-07 12:47:51 +01:00
kssenii
cc483c19a8 Fix merge with master 2023-12-07 12:14:49 +01:00
kssenii
8be3c9d218 Merge remote-tracking branch 'origin/master' into allow-to-change-some-cache-settings-without-restart 2023-12-07 12:14:24 +01:00
kssenii
614da21144 Better 2023-12-07 12:12:10 +01:00
Nikolay Degterinsky
995b51ef73 Allow avoiding resolving hostnames in DDLWorker 2023-12-07 03:43:26 +00:00
凌涛
d0675488ac rebase 2023-12-07 11:14:55 +08:00
Bharat Nallan Chakravarthy
029a0c0b77 fix clang-tidy 2023-12-06 12:30:39 -08:00
kssenii
f44f7c8c28 Allow to change some cache settings without server restart 2023-12-06 19:29:18 +01:00
Bharat Nallan Chakravarthy
6125c1692b only account for active parts 2023-12-06 08:43:30 -08:00
vdimir
f29386de0b
Materialize block in HashJoin for Type::EMPTY 2023-12-06 11:17:29 +00:00
Igor Nikonov
b4922eece4 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-06 10:44:30 +00:00
Nikolay Degterinsky
ed618d3be1 Fix crash in clickhouse-local 2023-12-06 04:24:43 +00:00
Bharat Nallan Chakravarthy
d1c5163a38 asynchronous metrics - total primary key bytes in memory 2023-12-05 18:46:16 -08:00
Igor Nikonov
f123dd0070 Fix addShard() 2023-12-05 18:07:51 +00:00
MikhailBurdukov
c41511e51c ignore_on_cluster_for_grants 2023-12-05 15:11:32 +00:00
Maksim Kita
c976f2fdf7 JIT fix arguments aliasing 2023-12-05 18:05:20 +03:00
Anton Popov
913ae1d267 Merge remote-tracking branch 'upstream/master' into HEAD 2023-12-05 14:47:32 +00:00
Anton Popov
c0e45c15fb add test for lightweight deletes and mutations 2023-12-05 13:38:25 +00:00
Nikita Mikhaylov
04d167c6d9 Better 2023-12-05 13:34:37 +01:00
Igor Nikonov
b2a269e8ce Formatting 2023-12-05 12:31:24 +00:00
Igor Nikonov
87a4c0bde4 Simple cleanup in distributed (while dealing with parallel replicas) 2023-12-05 12:25:02 +00:00
Igor Nikonov
71204c2a82 Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-12-05 12:00:45 +00:00
Alexey Milovidov
cbea3545aa
Revert "add function getClientHTTPHeader" 2023-12-05 13:34:34 +03:00
Alexey Milovidov
1f752b79db
Merge pull request #55894 from bharatnc/ncb/hostname-system-log-tables 
add hostname column to system log tables
 2023-12-05 06:42:43 +01:00
凌涛
9e05e79d66
Merge branch 'master' into feature/more_warnings 2023-12-05 11:50:05 +08:00
凌涛
6794bbe196 rebase master 2023-12-05 11:42:25 +08:00
凌涛
5a51fdac95 modify 2023-12-05 11:33:40 +08:00
Anton Popov
69896d94be fix tests 2023-12-04 21:40:19 +00:00
Anton Popov
40a1744b7c Merge remote-tracking branch 'upstream/master' into HEAD 2023-12-04 18:10:43 +00:00
kssenii
4a28f10c3d Minor cache changes 2023-12-04 19:02:37 +01:00
Alexey Milovidov
02439eee58
Merge pull request #49351 from ClickHouse/async-loader-integration 
Asynchronous loading of tables
 2023-12-04 17:16:12 +01:00
Mikhail Koviazin
bef3fff731
Add support for read-only mode in ZooKeeper 
This commit enables the read-only flag when connecting to the ZooKeeper server.

This flag is enabled by sending one extra byte when connecting,
and then receiving one extra byte during the first response.

In addition to that, we modify createIfNotExists to not complain
about attempting to alter a read-only ZooKeeper cluster if the node
already exists.

This makes ClickHouse more useful in the event of a loss of quorum,
user credentials are still accessible, which makes it possible to
connect to the cluster and run read queries.

Any DDL or DML query on a Distributed database or ReplicatedMergeTree
table will correctly fail, since it needs to write to ZooKeeper to
execute the query.

Any non-distributed query will be possible, which is ok since the
query was never replicated in the first place, there is no loss of
consistency.

Fixes #53749 as it seems to be the only thing 3.9 enforced.
 2023-12-04 16:47:12 +02:00
vdimir
a4ae90de0d
Merge pull request #57275 from ClickHouse/vdimir/merge_task_tmp_data 
Background merges correctly use temporary data storage in the cache
 2023-12-04 14:52:20 +01:00
Anton Popov
8d30e22a09 fix lighweight delete with heavy delete 2023-12-04 13:00:50 +00:00
Kseniia Sumarokova
715ec0d334
Merge pull request #56383 from ClickHouse/fix-backup-restore-with-mv 
Fix restore from backup with mat view and dropped source table
 2023-12-04 13:25:13 +01:00
Kseniia Sumarokova
4d3fde3624
Merge pull request #57424 from ClickHouse/fix-cache-background-download-queue-limit 
fs cache: add limit for background download
 2023-12-04 11:03:20 +01:00
Bharat Nallan Chakravarthy
440dc66a5c Merge remote-tracking branch 'upstream/master' into ncb/hostname-system-log-tables 2023-12-03 15:19:47 -08:00
Bharat Nallan Chakravarthy
efbf39ea06 improve TableNameHints 2023-12-02 00:45:21 -08:00
Igor Nikonov
3fec1f4110 Fix clang-tidy warning: move from moved object 2023-12-01 21:01:35 +00:00
Anton Popov
7fbe7462b6 add mutation command to apply deleted mask 2023-12-01 19:12:05 +00:00
JackyWoo
8a68d51261
Add index to table system.numbers (#50909) 2023-12-01 19:59:25 +01:00
robot-ch-test-poll
1b49463bd2
Merge pull request #55841 from nickitat/optimize_reading3 
Optimize reading from cache
 2023-12-01 17:36:57 +01:00
Nikolai Kochetov
0b4131546a
Merge pull request #56813 from jsc0218/SystemTablesFilterEngine 
Able to Filter Engine When Scanning System Tables
 2023-12-01 16:02:27 +01:00
kssenii
60d7a268ce Add a limit for background download queue 2023-12-01 14:53:31 +01:00
Maksim Kita
8b5e6af689 Fixed tests 2023-12-01 13:45:57 +03:00
Maksim Kita
8bc7dbc8f7 Enable JIT compilation for aggregation without key 2023-12-01 13:45:57 +03:00
Julia Kartseva
1f8031c6e1
Merge pull request #56892 from icuken/alter_temporary_table 
allow ALTER for TEMPORARY tables
 2023-11-30 18:29:43 -08:00
Yakov Olkhovskiy
416d28687e Merge remote-tracking branch 'origin/master' into feature-server-iface-metrics 2023-11-30 23:00:04 +00:00
Igor Nikonov
45ebe78d32 Cleanup 2023-11-30 21:40:45 +00:00
Yakov Olkhovskiy
549a90ba09 unfortunately we can't release session's named_session early because its context's ClientInfo still may be used in ~Session - next session can start and compete for this data 2023-11-30 19:56:25 +00:00
Raúl Marín
46acd8736c
Merge pull request #57189 from Algunenano/fix_respect_nulls 
Fix RESPECT NULLS
 2023-11-30 20:21:42 +01:00
Sergey Kviatkevich
afae432e99 allow ALTER for TEMPORARY table 2023-11-30 21:42:12 +03:00
Yakov Olkhovskiy
d2bfddd28d attempt to solve data race, small optimization 2023-11-30 18:29:00 +00:00
Igor Nikonov
b4bf611d5d Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-11-30 14:09:26 +00:00
Igor Nikonov
59daabe95a Some refactoring and cleanup 
+ change load balancing for custom key w/o hedged connections
  (need the test though)
 2023-11-30 14:08:01 +00:00
Raúl Marín
a331e346a1 Style fix 2023-11-30 14:20:32 +01:00
kssenii
7bd458ac91 Merge remote-tracking branch 'origin/master' into fix-backup-restore-with-mv 2023-11-30 14:16:11 +01:00
Raúl Marín
7fa626bfef Reject NULL actions on normal functions and fix serialization 2023-11-30 13:16:51 +01:00
Alexander Gololobov
b9b94a8e0a
Merge pull request #57358 from ClickHouse/shutdown_order 
Shutdown disks after tables
 2023-11-30 09:13:29 +01:00
凌涛
d9f4b4d2c0 support parts, tables, databases 2023-11-30 10:56:31 +08:00
Alexander Tokmakov
12cb3b3b87 Merge branch 'master' into drop_database_replica_no_throw 2023-11-30 00:00:07 +01:00
Alexander Gololobov
2ccd7237f5 Shutdown disks after tables 2023-11-29 20:04:37 +01:00
Alexander Tokmakov
a8ed97ac39
Update DDLTask.cpp 2023-11-29 19:51:32 +01:00
Alexander Tokmakov
a2dc2d9a53
Mark a setting obsolete (#57336) 
* Update Settings.h

* Update InterpreterUndropQuery.cpp

* Update 02888_obsolete_settings.reference

* Update test.py

* Update 02681_undrop_query.sql

* Update 02681_undrop_query_uuid.sh
 2023-11-29 19:24:11 +01:00
kssenii
296afceed8 Merge remote-tracking branch 'origin/master' into fix-backup-restore-with-mv 2023-11-29 18:06:09 +01:00
Alexander Tokmakov
97b63bbbca
Merge pull request #57339 from ClickHouse/fix_ddlworker_hostname_change 
Ignore finished ON CLUSTER tasks if hostname changed
 2023-11-29 17:46:17 +01:00
vdimir
b5babe1692
MergeTask uses temporary data storage 2023-11-29 16:18:32 +00:00
Han Fei
4c0efb0262
Merge pull request #53240 from hanfei1991/hanfei/statistic 
use statistic to order prewhere conditions better
 2023-11-29 12:26:44 +01:00
Kseniia Sumarokova
4d27f87be8
Merge pull request #48425 from ClickHouse/vdimir/enable-tmp-in-cache-ci 
Enable temporary_data_in_cache in s3 tests in CI
 2023-11-29 11:59:32 +01:00
vdimir
470a092e3d
Merge pull request #51745 from ClickHouse/vdimir/setting_join_algorithm_order 
Setting JoinAlgorithm respect specified order
 2023-11-29 11:24:39 +01:00
Bharat Nallan Chakravarthy
4ca3d0d5c4 better hints when table doesn't exist 2023-11-28 21:34:02 -08:00
Alexander Tokmakov
84abbafcc5 ignore finished ddl tasks if hostname changed 2023-11-29 02:06:52 +01:00
Alexander Tokmakov
67f0a0e3e4 don't throw if noop when dropping db replica in batch 2023-11-28 23:57:55 +01:00
vdimir
a6eb2ad930
Fix JoinStep::describeJoinActions 2023-11-28 19:09:01 +00:00
Igor Nikonov
a06a71834a Merge remote-tracking branch 'origin/master' into pr-custom-key-failover 2023-11-28 18:07:04 +00:00
Igor Nikonov
25eee6cd01 Fix clang-tidy warnings 2023-11-28 18:06:13 +00:00
Nikolai Kochetov
84194f10f2 Fixing tests. 2023-11-28 18:04:30 +00:00
Nikita Taranov
03450d5077 merge fixes 2023-11-28 18:24:05 +01:00
Nikita Taranov
52f644c0df Merge branch 'master' into optimize_reading3 2023-11-28 16:36:38 +01:00
Han Fei
6733438adc Merge branch 'master' into hanfei/statistic 2023-11-28 16:35:00 +01:00
János Benjamin Antal
4c2451f837
Merge pull request #56741 from ClickHouse/well-known-protos 2023-11-28 15:21:15 +01:00
Raúl Marín
48fff248e5 Merge remote-tracking branch 'blessed/master' into fix_respect_nulls 2023-11-28 12:28:36 +01:00
Raúl Marín
335e7bf7ef
Merge pull request #57183 from bigo-sg/opt_array_join 
Trivial improvement on array join
 2023-11-28 12:21:00 +01:00
Raúl Marín
fe1bbf9726
Merge pull request #57258 from Algunenano/cleanup_interval 
Remove addBatchSinglePlaceFromInterval
 2023-11-28 12:02:12 +01:00
vdimir
aa5d973204
Fix direct join with range dictionary in 
JoinedTables.cpp and update test queries
 2023-11-28 11:01:38 +00:00
vdimir
5c20e5039b
Remove redundant descriptionForPlan 2023-11-28 11:01:38 +00:00
vdimir
af81cb8252
Setting JoinAlgorithm respect specified order 2023-11-28 11:01:35 +00:00