Commit Graph

14006 Commits

Author SHA1 Message Date
Victor Krasnov
258f9927b4 Merge branch 'master' into ADQM-987 2023-08-22 16:49:32 +03:00
Robert Schulze
acaa2eee58
Merge pull request #53556 from ClibMouse/feature/big-endian-UUID-correction
Change Big Endian-UUID to work the same as Little Endian-UUID
2023-08-22 11:57:44 +02:00
alesapin
1a5a9fca87
Merge pull request #53605 from irenjj/feat_53538
Add column 'name' to system.clusters as an alias to cluster.
2023-08-22 11:34:21 +02:00
Robert Schulze
a20ceef398
Merge pull request #53447 from davvard/master
USearch HNSW index for ANN search
2023-08-21 23:34:26 +02:00
Michael Kolupaev
2f4d433e69 Parquet filter pushdown 2023-08-21 14:15:52 -07:00
Antonio Andelic
ed15babddd
Merge pull request #53198 from ClickHouse/improve-archive-reading
Improve reading from archives
2023-08-21 14:11:25 +02:00
Victor Krasnov
a7813260c0 Merge branch 'master' into ADQM-987 2023-08-21 14:45:13 +03:00
Robert Schulze
2c330516b8
Another compile fix 2023-08-21 11:12:39 +00:00
irenjj
a9b65ee929 Add Alias for system.clusters. 2023-08-20 14:47:07 +08:00
irenjj
71459c6d65 Revert 2023-08-20 14:35:59 +08:00
irenjj
e64a1fc26d Add column 'name' to system.clusters as an alias to cluster. 2023-08-20 11:52:14 +08:00
Igor Nikonov
a69ea02418 Merge remote-tracking branch 'origin/master' into fix-parallel-replicas-multiply-result 2023-08-19 21:43:25 +00:00
Kseniia Sumarokova
c0b8d7eddf
Merge pull request #50999 from ilejn/system_kafka_consumers
system.kafka_consumers table to monitor kafka consumers
2023-08-19 13:31:18 +04:00
Alexey Milovidov
8520a7baca
Merge pull request #53559 from azat/skip-index-set-fix
Fix possible UB in Set skipping index for functions with incorrect args
2023-08-19 02:06:18 +03:00
Alexey Milovidov
6fb24fe74d
Merge pull request #53560 from azat/skip-index-inverted-fix-ub
Fix possible UB in inverted indexes (experimental feature)
2023-08-19 02:06:04 +03:00
Igor Nikonov
aebda10f7c Merge remote-tracking branch 'origin/master' into fix-parallel-replicas-multiply-result 2023-08-18 23:00:58 +00:00
Alexey Milovidov
7ad8ea78bb
Merge pull request #48990 from incfly/partition
Add partition to the system part_log and merge table.
2023-08-18 21:38:53 +03:00
Azat Khuzhin
c78b94bed6 Fix possible UB in inverted indexes (experimental feature)
It is possible to have incorrect types there in case of index contains
functions, add a proper check after value had been changed.

<details>

<summary>MSan report</summary>

```
==182==WARNING: MemorySanitizer: use-of-uninitialized-value
    0 0x55658547c59e in DB::MergeTreeConditionInverted::traverseASTEquals() build_docker/./src/Storages/MergeTree/MergeTreeIndexInverted.cpp:551:26
    1 0x556585475566 in DB::MergeTreeConditionInverted::traverseAtomAST() build_docker/./src/Storages/MergeTree/MergeTreeIndexInverted.cpp:438:21
    2 0x55658547e73e in DB::MergeTreeConditionInverted::MergeTreeConditionInverted()::$_1::operator()(DB::RPNBuilderTreeNode const&, DB::MergeTreeConditionInverted::RPNElement&) const build_docker/./src/Storages/MergeTree/MergeTreeIndexInverted.cpp:228:73
    9 0x556585489e35 in DB::RPNBuilder<DB::MergeTreeConditionInverted::RPNElement>::traverseTree(DB::RPNBuilderTreeNode const&) build_docker/./src/Storages/MergeTree/RPNBuilder.h:252:14
    10 0x556585489921 in DB::RPNBuilder<DB::MergeTreeConditionInverted::RPNElement>::traverseTree(DB::RPNBuilderTreeNode const&) build_docker/./src/Storages/MergeTree/RPNBuilder.h:239:21
    11 0x556585489921 in DB::RPNBuilder<DB::MergeTreeConditionInverted::RPNElement>::traverseTree(DB::RPNBuilderTreeNode const&) build_docker/./src/Storages/MergeTree/RPNBuilder.h:239:21
    12 0x5565854804e6 in DB::RPNBuilder<DB::MergeTreeConditionInverted::RPNElement>::RPNBuilder(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, DB::Block, std::__1::shared_ptr<DB::PreparedSets>, std::__1::function<bool (DB::RPNBuilderTreeNode const&, DB::MergeTreeConditionInverted::RPNElement&)> const&) build_docker/./src/Storages/MergeTree/RPNBuilder.h:218:9
    13 0x55658546fb87 in DB::MergeTreeConditionInverted::MergeTreeConditionInverted(DB::SelectQueryInfo const&, std::__1::shared_ptr<DB::Context const>, DB::Block const&, DB::GinFilterParameters const&, DB::ITokenExtractor const*) build_docker/./src/Storages/MergeTree/MergeTreeIndexInverted.cpp:223:28
    19 0x55658547ceda in DB::MergeTreeIndexInverted::createIndexCondition(DB::SelectQueryInfo const&, std::__1::shared_ptr<DB::Context const>) const build_docker/./src/Storages/MergeTree/MergeTreeIndexInverted.cpp:716:12
    20 0x556587125734 in DB::buildIndexes(std::__1::optional<DB::ReadFromMergeTree::Indexes>&, std::__1::shared_ptr<DB::ActionsDAG>, DB::MergeTreeData const&, std::__1::shared_ptr<DB::Context const> const&, DB::SelectQueryInfo const&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&) build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:1292:48
    21 0x556587121aaa in DB::ReadFromMergeTree::applyFilters() build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:1305:5
    22 0x55658726cf4f in DB::QueryPlanOptimizations::optimizeTreeThirdPass(DB::QueryPlan::Node&, std::__1::list<DB::QueryPlan::Node, std::__1::allocator<DB::QueryPlan::Node>>&) build_docker/./src/Processors/QueryPlan/Optimizations/optimizeTree.cpp:201:38
    23 0x5565870bc489 in DB::QueryPlan::optimize(DB::QueryPlanOptimizationSettings const&) build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:485:5
    24 0x5565870b919a in DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:161:5
    25 0x55658206385a in DB::InterpreterSelectWithUnionQuery::execute() build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:379:31
    26 0x556582c15c50 in DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) build_docker/./src/Interpreters/executeQuery.cpp:751:40
    27 0x556582c09681 in DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) build_docker/./src/Interpreters/executeQuery.cpp:1173:30
    28 0x5565860e7bd7 in DB::TCPHandler::runImpl() build_docker/./src/Server/TCPHandler.cpp:421:24
    29 0x556586126e9e in DB::TCPHandler::run() build_docker/./src/Server/TCPHandler.cpp:2057:9
```

</details>

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-08-18 18:15:17 +02:00
Azat Khuzhin
1e2b0c8dc4 Fix possible UB in Set skipping index for functions with incorrect args
Usually functions are validated before index analysis, however it is
not the case for parallel replicas, and it requires additional checks
before interpreting function arguments.

<details>

<summary>stack trace</summary>

```
==172==WARNING: MemorySanitizer: use-of-uninitialized-value
    0 0x55cf82aedd6c in DB::ASTFunction* typeid_cast<DB::ASTFunction*, DB::IAST>(DB::IAST*) build_docker/./src/Common/typeid_cast.h:42:73
    1 0x55cf82aedd6c in DB::TypePromotion<DB::IAST>::CastHelper<DB::ASTFunction, false, false>::value(DB::IAST*) build_docker/./src/Common/TypePromotion.h:38:43
    2 0x55cf82aedd6c in std::__1::invoke_result<decltype(&CastHelper<DB::ASTFunction, false>::value), DB::TypePromotion<DB::IAST>::CastHelper<DB::ASTFunction, false, std::is_reference_v<DB::ASTFunction>>, DB::IAST*>::type DB::TypePromotion<DB::IAST>::as<DB::ASTFunction>() build_docker/./src/>
    3 0x55cf82aedd6c in DB::MergeTreeIndexConditionSet::operatorFromAST(std::__1::shared_ptr<DB::IAST>&) build_docker/./src/Storages/MergeTree/MergeTreeIndexSet.cpp:602:25
    4 0x55cf82ae5bc1 in DB::MergeTreeIndexConditionSet::traverseAST(std::__1::shared_ptr<DB::IAST>&) const build_docker/./src/Storages/MergeTree/MergeTreeIndexSet.cpp:547:9
    5 0x55cf82ae5de6 in DB::MergeTreeIndexConditionSet::traverseAST(std::__1::shared_ptr<DB::IAST>&) const build_docker/./src/Storages/MergeTree/MergeTreeIndexSet.cpp:552:13
    6 0x55cf82ae06db in DB::MergeTreeIndexConditionSet::MergeTreeIndexConditionSet() build_docker/.>
    ...
    12 0x55cf82aef09c in DB::MergeTreeIndexSet::createIndexCondition() const build_docker/./src/Storages/MergeTree/MergeTreeIndexSet.cpp:703:12
    13 0x55cf84951cd4 in DB::buildIndexes()
    14 0x55cf84955ed3 in DB::ReadFromMergeTree::selectRangesToReadImpl()
    15 0x55cf8494caef in DB::ReadFromMergeTree::selectRangesToRead()
    16 0x55cf82a409a9 in DB::MergeTreeDataSelectExecutor::estimateNumMarksToRead()
    17 0x55cf827f728d in DB::MergeTreeData::canUseParallelReplicasBasedOnPKAnalysis()
    18 0x55cf827f627e in DB::MergeTreeData::getQueryProcessingStage()
    19 0x55cf7f2f4969 in DB::InterpreterSelectQuery::getSampleBlockImpl() build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:937:31
    20 0x55cf7f2daa00 in DB::InterpreterSelectQuery::InterpreterSelectQuery()
    24 0x55cf7f520b98 in DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter()
    25 0x55cf7f51b6cd in DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery()
    27 0x55cf7f1d4ea9 in DB::InterpreterFactory::get(std::__1::shared_ptr<DB::IAST>&, std::__1::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) build_docker/./src/Interpreters/InterpreterFactory.cpp:162:16
    28 0x55cf8012e485 in DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) build_docker/./src/Interpreters/executeQuery.cpp:1032:31
    29 0x55cf80121bc1 in DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) build_docker/./src/Interpreters/executeQuery.cpp:1229:30
    30 0x55cf8389295f in DB::TCPHandler::runImpl() build_docker/./src/Server/TCPHandler.cpp:424:24
    31 0x55cf838d7dfb in DB::TCPHandler::run() build_docker/./src/Server/TCPHandler.cpp:2050:9
```

</details>

CI: https://s3.amazonaws.com/clickhouse-test-reports/53214/d99b10c340909ed4ee2e6edf0921e8a2f8561b0d/fuzzer_astfuzzermsan/report.html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-08-18 18:02:46 +02:00
Austin Kothig
6b42975d33
Change BE-UUID to work the same as LE-UUID. Included high and low getters to provide cleaner code when accessing undertype. 2023-08-18 08:19:46 -07:00
Yakov Olkhovskiy
84cb358c14
Merge pull request #43210 from amosbird/storage_executable_stderr
Add different reactions when external command stderr has data
2023-08-18 11:17:46 -04:00
Alexander Tokmakov
a6684f7a37
Merge pull request #53484 from azat/rmt-queue-cleanup
Remove outdated code in ReplicatedMergeTreeQueue::initialize()
2023-08-18 18:11:33 +03:00
Alexander Gololobov
50061b9cb4
Merge pull request #53452 from ClickHouse/such_much_watch
Deduplicate same watch callbacks registered multiple times
2023-08-18 14:43:03 +02:00
Alexander Tokmakov
0bd378afd9
Merge pull request #53381 from bakwc/bakwc-patch-2
Fixed system.data_skipping_indices for MaterializedMySQL
2023-08-18 14:22:40 +03:00
Amos Bird
b6de70b042
StorageExecutable check_exit_code default to false 2023-08-18 15:38:47 +08:00
Amos Bird
c43bf153f5
Refactor 2023-08-18 15:38:46 +08:00
Amos Bird
dd0c71b32a
Add error_exit_reaction 2023-08-18 15:38:46 +08:00
Amos Bird
476f3cedc1
Various reactions when executable stderr has data 2023-08-18 15:38:45 +08:00
Alexey Milovidov
a882bf1bb8
Merge pull request #53495 from ClickHouse/i_hate_materialized_views
Fix a race condition between RESTART REPLICAS and DROP DATABASE
2023-08-18 06:40:04 +03:00
Igor Nikonov
06c98956b3 Merge remote-tracking branch 'origin/master' into fix-parallel-replicas-multiply-result 2023-08-17 20:42:50 +00:00
Jianfei Hu
0c81546960 remove the comments.
Signed-off-by: Jianfei Hu <hujianfei258@gmail.com>
2023-08-17 20:13:33 +00:00
pufit
a52249872e
Merge pull request #53313 from jorisgio/joris/max_threads_for_indexes
add max_threads_for_indexes settings
2023-08-17 13:59:53 -04:00
Alexander Gololobov
54e2be4e32 Use WatchCallbackPtr 2023-08-17 17:32:04 +02:00
Alexander Tokmakov
89c365803f
Merge pull request #53424 from ClickHouse/exception_message_patterns6
Less exceptions with runtime format string
2023-08-17 17:58:04 +03:00
Igor Nikonov
3e9ae1751e Merge remote-tracking branch 'origin/master' into fix-parallel-replicas-multiply-result 2023-08-17 14:45:37 +00:00
Filipp Ozinov
ce87451b66
Merge branch 'ClickHouse:master' into bakwc-patch-2 2023-08-17 18:37:00 +04:00
Alexander Tokmakov
72b52250ea
Merge pull request #53501 from azat/dist-flush_on_detach
Add ability to turn off flush of Distributed on DETACH/DROP/server shutdown
2023-08-17 16:56:04 +03:00
Azat Khuzhin
fcb66379d1 Remove outdated code in ReplicatedMergeTreeQueue::initialize()
Initially added in #28817
But got cleaned up in #41981

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-08-17 13:45:26 +02:00
Robert Schulze
ec7daed52f
Merge pull request #50276 from ClibMouse/feature/mergetree-checksum-big-endian-support
MergeTree & SipHash checksum big-endian support
2023-08-17 13:27:18 +02:00
Robert Schulze
42644bfbf5
Rename file to MergeTreeIndexUSearch.h/cpp 2023-08-17 09:54:12 +00:00
Robert Schulze
f420006e09
Make the code consistent with Annoy index and fix build 2023-08-17 09:42:13 +00:00
Robert Schulze
456a02da1b
Merge remote-tracking branch 'rschu1ze/master' 2023-08-17 09:30:27 +00:00
Robert Schulze
1215a23de6
Merge pull request #53422 from rschu1ze/consistency-fixes-for-caches
Refactorings for configuration of in-memory caches
2023-08-17 11:08:32 +02:00
Azat Khuzhin
17ca2661a1 Add ability to turn off flush of Distributed on DETACH/DROP/server shutdown
Sometimes you can have tons of data there, i.e. few TiBs, and sending
them on server shutdown does not looks sane (maybe there is a bug and
you need to update/restart to fix flushing).

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-08-17 08:58:06 +02:00
Igor Nikonov
49f7b4acd6 Merge remote-tracking branch 'origin/master' into fix-parallel-replicas-multiply-result 2023-08-16 23:31:49 +00:00
Igor Nikonov
21ef1f1d1c Remove unnecessary Scalars usage for cluster_for_parallel_replicas 2023-08-16 23:24:01 +00:00
Alexander Tokmakov
d7272410bd the worst fix for a deadlock ever 2023-08-17 00:42:51 +02:00
Robert Schulze
8a58e0df3e
Merge remote-tracking branch 'rschu1ze/master' into consistency-fixes-for-caches 2023-08-16 21:26:04 +00:00
Robert Schulze
353854520c
Merge remote-tracking branch 'rschu1ze/master' into factorize-constants 2023-08-16 21:24:21 +00:00
Alexander Tokmakov
69bc4f5139 Merge branch 'master' into exception_message_patterns6 2023-08-16 22:58:24 +02:00