Commit Graph

13913 Commits

Author SHA1 Message Date
Azat Khuzhin
1e2b0c8dc4 Fix possible UB in Set skipping index for functions with incorrect args
Usually functions are validated before index analysis, however it is
not the case for parallel replicas, and it requires additional checks
before interpreting function arguments.

<details>

<summary>stack trace</summary>

```
==172==WARNING: MemorySanitizer: use-of-uninitialized-value
    0 0x55cf82aedd6c in DB::ASTFunction* typeid_cast<DB::ASTFunction*, DB::IAST>(DB::IAST*) build_docker/./src/Common/typeid_cast.h:42:73
    1 0x55cf82aedd6c in DB::TypePromotion<DB::IAST>::CastHelper<DB::ASTFunction, false, false>::value(DB::IAST*) build_docker/./src/Common/TypePromotion.h:38:43
    2 0x55cf82aedd6c in std::__1::invoke_result<decltype(&CastHelper<DB::ASTFunction, false>::value), DB::TypePromotion<DB::IAST>::CastHelper<DB::ASTFunction, false, std::is_reference_v<DB::ASTFunction>>, DB::IAST*>::type DB::TypePromotion<DB::IAST>::as<DB::ASTFunction>() build_docker/./src/>
    3 0x55cf82aedd6c in DB::MergeTreeIndexConditionSet::operatorFromAST(std::__1::shared_ptr<DB::IAST>&) build_docker/./src/Storages/MergeTree/MergeTreeIndexSet.cpp:602:25
    4 0x55cf82ae5bc1 in DB::MergeTreeIndexConditionSet::traverseAST(std::__1::shared_ptr<DB::IAST>&) const build_docker/./src/Storages/MergeTree/MergeTreeIndexSet.cpp:547:9
    5 0x55cf82ae5de6 in DB::MergeTreeIndexConditionSet::traverseAST(std::__1::shared_ptr<DB::IAST>&) const build_docker/./src/Storages/MergeTree/MergeTreeIndexSet.cpp:552:13
    6 0x55cf82ae06db in DB::MergeTreeIndexConditionSet::MergeTreeIndexConditionSet() build_docker/.>
    ...
    12 0x55cf82aef09c in DB::MergeTreeIndexSet::createIndexCondition() const build_docker/./src/Storages/MergeTree/MergeTreeIndexSet.cpp:703:12
    13 0x55cf84951cd4 in DB::buildIndexes()
    14 0x55cf84955ed3 in DB::ReadFromMergeTree::selectRangesToReadImpl()
    15 0x55cf8494caef in DB::ReadFromMergeTree::selectRangesToRead()
    16 0x55cf82a409a9 in DB::MergeTreeDataSelectExecutor::estimateNumMarksToRead()
    17 0x55cf827f728d in DB::MergeTreeData::canUseParallelReplicasBasedOnPKAnalysis()
    18 0x55cf827f627e in DB::MergeTreeData::getQueryProcessingStage()
    19 0x55cf7f2f4969 in DB::InterpreterSelectQuery::getSampleBlockImpl() build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:937:31
    20 0x55cf7f2daa00 in DB::InterpreterSelectQuery::InterpreterSelectQuery()
    24 0x55cf7f520b98 in DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter()
    25 0x55cf7f51b6cd in DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery()
    27 0x55cf7f1d4ea9 in DB::InterpreterFactory::get(std::__1::shared_ptr<DB::IAST>&, std::__1::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) build_docker/./src/Interpreters/InterpreterFactory.cpp:162:16
    28 0x55cf8012e485 in DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) build_docker/./src/Interpreters/executeQuery.cpp:1032:31
    29 0x55cf80121bc1 in DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) build_docker/./src/Interpreters/executeQuery.cpp:1229:30
    30 0x55cf8389295f in DB::TCPHandler::runImpl() build_docker/./src/Server/TCPHandler.cpp:424:24
    31 0x55cf838d7dfb in DB::TCPHandler::run() build_docker/./src/Server/TCPHandler.cpp:2050:9
```

</details>

CI: https://s3.amazonaws.com/clickhouse-test-reports/53214/d99b10c340909ed4ee2e6edf0921e8a2f8561b0d/fuzzer_astfuzzermsan/report.html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-08-18 18:02:46 +02:00
Alexander Tokmakov
0bd378afd9
Merge pull request #53381 from bakwc/bakwc-patch-2
Fixed system.data_skipping_indices for MaterializedMySQL
2023-08-18 14:22:40 +03:00
Alexey Milovidov
a882bf1bb8
Merge pull request #53495 from ClickHouse/i_hate_materialized_views
Fix a race condition between RESTART REPLICAS and DROP DATABASE
2023-08-18 06:40:04 +03:00
pufit
a52249872e
Merge pull request #53313 from jorisgio/joris/max_threads_for_indexes
add max_threads_for_indexes settings
2023-08-17 13:59:53 -04:00
Alexander Tokmakov
89c365803f
Merge pull request #53424 from ClickHouse/exception_message_patterns6
Less exceptions with runtime format string
2023-08-17 17:58:04 +03:00
Filipp Ozinov
ce87451b66
Merge branch 'ClickHouse:master' into bakwc-patch-2 2023-08-17 18:37:00 +04:00
Alexander Tokmakov
72b52250ea
Merge pull request #53501 from azat/dist-flush_on_detach
Add ability to turn off flush of Distributed on DETACH/DROP/server shutdown
2023-08-17 16:56:04 +03:00
Robert Schulze
ec7daed52f
Merge pull request #50276 from ClibMouse/feature/mergetree-checksum-big-endian-support
MergeTree & SipHash checksum big-endian support
2023-08-17 13:27:18 +02:00
Robert Schulze
1215a23de6
Merge pull request #53422 from rschu1ze/consistency-fixes-for-caches
Refactorings for configuration of in-memory caches
2023-08-17 11:08:32 +02:00
Azat Khuzhin
17ca2661a1 Add ability to turn off flush of Distributed on DETACH/DROP/server shutdown
Sometimes you can have tons of data there, i.e. few TiBs, and sending
them on server shutdown does not looks sane (maybe there is a bug and
you need to update/restart to fix flushing).

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-08-17 08:58:06 +02:00
Alexander Tokmakov
d7272410bd the worst fix for a deadlock ever 2023-08-17 00:42:51 +02:00
Robert Schulze
8a58e0df3e
Merge remote-tracking branch 'rschu1ze/master' into consistency-fixes-for-caches 2023-08-16 21:26:04 +00:00
Robert Schulze
353854520c
Merge remote-tracking branch 'rschu1ze/master' into factorize-constants 2023-08-16 21:24:21 +00:00
Alexander Tokmakov
69bc4f5139 Merge branch 'master' into exception_message_patterns6 2023-08-16 22:58:24 +02:00
Alexander Tokmakov
b32bdd1b0a
Merge pull request #53164 from ClickHouse/fixes_for_detached_parts
Fixes for detach/attach partition and broken detached parts cleanup
2023-08-16 00:09:13 +03:00
Robert Schulze
56a6d20137
Merge remote-tracking branch 'rschu1ze/master' into factorize-constants 2023-08-15 18:21:25 +00:00
Robert Schulze
f558048bea
Merge remote-tracking branch 'rschu1ze/master' into consistency-fixes-for-caches 2023-08-15 17:12:52 +00:00
Alexander Tokmakov
83c10dad18 Merge branch 'master' into exception_message_patterns6 2023-08-15 15:20:09 +02:00
Alexander Tokmakov
bf40767f10 fix another race 2023-08-15 14:27:49 +02:00
Kruglov Pavel
cac2628b41
Merge pull request #53426 from ClickHouse/analyzer-fix-virtuals-distr
Analyzer: fix virtual columns in StorageDistributed
2023-08-15 13:55:42 +02:00
Robert Schulze
8992cc5af4
Factorize constants 2023-08-15 08:56:40 +00:00
Robert Schulze
f317903073
Merge branch 'ClickHouse:master' into consistency-fixes-for-caches 2023-08-15 10:25:08 +02:00
Alexander Tokmakov
376202f739 fix creation of empty parts 2023-08-15 00:26:17 +02:00
Alexander Tokmakov
894ec8e0d0 less exceptions with runtime format string 2023-08-14 21:45:59 +02:00
Dmitry Novik
1244828555 Analyzer: fix virtual columns in StorageDistributed 2023-08-14 19:27:05 +02:00
Alexander Tokmakov
f0f2cd8c13 Merge branch 'master' into fixes_for_detached_parts 2023-08-14 19:20:45 +02:00
ltrk2
2846ea49b4
Merge branch 'master' into feature/mergetree-checksum-big-endian-support 2023-08-14 13:02:30 -04:00
Robert Schulze
3472a928d8
Cosmetics: Rename dropCache() to clearCaches()
"dropCache()" is a bad name, it sounds like that the cache can no longer
be used after the call. Not true, we only "clear" the cache.
2023-08-14 15:42:32 +00:00
Filipp Ozinov
5a180b53ed
Style fix 2023-08-14 12:30:29 +04:00
Alexey Milovidov
9656bd2738
Merge pull request #53372 from ClickHouse/Fix_lazy_database_and_system_table_query
Fix loading lazy database during system.table select query
2023-08-14 08:03:19 +03:00
Yakov Olkhovskiy
300399d1f5
Merge pull request #52329 from ClibMouse/feature/mergetree-checksum-on-big-endian
Implement endianness-indepedent support for MergeTree checksums
2023-08-13 23:59:05 -04:00
Filipp Ozinov
0cccb2ac3b
Fixed system.data_skipping_indices for MaterializedMySQL
Related to #53350
2023-08-14 01:57:41 +04:00
Smita Kulkarni
c25bb44e55 Fixed style check 2023-08-13 21:45:33 +02:00
Smita Kulkarni
ee772b7376 Avoid loading tables from lazy database when not needed 2023-08-13 20:15:58 +02:00
Alexey Milovidov
17127cf23f
Merge pull request #53337 from hanfei1991/hanfei/improve_move_to_prewhere
improve `move_primary_key_columns_to_end_of_prewhere`
2023-08-12 17:45:30 +03:00
Han Fei
0b3f9918a3 fix style 2023-08-12 14:40:02 +02:00
robot-ch-test-poll
967067bbb8
Merge pull request #53250 from evillique/postgresql-single-quotes
Fix character escaping in the PostgreSQL engine
2023-08-12 13:15:49 +02:00
Alexey Milovidov
3c2dd4e543
Update MergeTreeWhereOptimizer.cpp 2023-08-12 03:40:19 +03:00
Alexey Milovidov
6176971b3f
Update StoragePostgreSQL.cpp 2023-08-12 03:33:58 +03:00
Alexey Milovidov
4b59d265e1
Merge pull request #49723 from DimasKovas/fix_storage_buffer_reschedule
Some unclear change in StorageBuffer::reschedule() for something
2023-08-12 03:28:23 +03:00
Alexey Milovidov
e5f33511c6
Merge pull request #53242 from evillique/alter-exception
Fix LOGICAL_ERROR exception in ALTER query
2023-08-12 03:18:32 +03:00
Han Fei
e70dc7d507 improve move_primary_key_columns_to_end_of_prewhere 2023-08-11 17:36:52 +02:00
Anton Popov
8744a0269a
Merge pull request #52530 from CurtizJ/fix-alter-delete-skip-index
Fix recalculation of skip indexes and projections in `ALTER DELETE` queries
2023-08-11 15:23:05 +02:00
ltrk2
1dc1b54c68
Merge branch 'master' into feature/mergetree-checksum-big-endian-support 2023-08-11 07:48:49 -04:00
ltrk2
94dbef1e50
Merge branch 'master' into feature/mergetree-checksum-on-big-endian 2023-08-11 07:47:48 -04:00
Joris Giovannangeli
602570b462
add max_threads for index settings 2023-08-11 10:20:07 +02:00
Alexey Milovidov
aae35f3b1c
Merge pull request #53184 from ClickHouse/better-ccache
Better usage of ccache
2023-08-11 04:43:45 +03:00
Alexander Tokmakov
5a8b8203b2 fix 2023-08-10 23:22:51 +02:00
Alexander Tokmakov
3d59ebe108 fix 2023-08-10 20:11:22 +02:00
Nikolay Degterinsky
4f0be777c5 Fix build 2023-08-10 16:39:43 +00:00