Commit Graph

122650 Commits

Author SHA1 Message Date
Azat Khuzhin
1e2b0c8dc4 Fix possible UB in Set skipping index for functions with incorrect args
Usually functions are validated before index analysis, however it is
not the case for parallel replicas, and it requires additional checks
before interpreting function arguments.

<details>

<summary>stack trace</summary>

```
==172==WARNING: MemorySanitizer: use-of-uninitialized-value
    0 0x55cf82aedd6c in DB::ASTFunction* typeid_cast<DB::ASTFunction*, DB::IAST>(DB::IAST*) build_docker/./src/Common/typeid_cast.h:42:73
    1 0x55cf82aedd6c in DB::TypePromotion<DB::IAST>::CastHelper<DB::ASTFunction, false, false>::value(DB::IAST*) build_docker/./src/Common/TypePromotion.h:38:43
    2 0x55cf82aedd6c in std::__1::invoke_result<decltype(&CastHelper<DB::ASTFunction, false>::value), DB::TypePromotion<DB::IAST>::CastHelper<DB::ASTFunction, false, std::is_reference_v<DB::ASTFunction>>, DB::IAST*>::type DB::TypePromotion<DB::IAST>::as<DB::ASTFunction>() build_docker/./src/>
    3 0x55cf82aedd6c in DB::MergeTreeIndexConditionSet::operatorFromAST(std::__1::shared_ptr<DB::IAST>&) build_docker/./src/Storages/MergeTree/MergeTreeIndexSet.cpp:602:25
    4 0x55cf82ae5bc1 in DB::MergeTreeIndexConditionSet::traverseAST(std::__1::shared_ptr<DB::IAST>&) const build_docker/./src/Storages/MergeTree/MergeTreeIndexSet.cpp:547:9
    5 0x55cf82ae5de6 in DB::MergeTreeIndexConditionSet::traverseAST(std::__1::shared_ptr<DB::IAST>&) const build_docker/./src/Storages/MergeTree/MergeTreeIndexSet.cpp:552:13
    6 0x55cf82ae06db in DB::MergeTreeIndexConditionSet::MergeTreeIndexConditionSet() build_docker/.>
    ...
    12 0x55cf82aef09c in DB::MergeTreeIndexSet::createIndexCondition() const build_docker/./src/Storages/MergeTree/MergeTreeIndexSet.cpp:703:12
    13 0x55cf84951cd4 in DB::buildIndexes()
    14 0x55cf84955ed3 in DB::ReadFromMergeTree::selectRangesToReadImpl()
    15 0x55cf8494caef in DB::ReadFromMergeTree::selectRangesToRead()
    16 0x55cf82a409a9 in DB::MergeTreeDataSelectExecutor::estimateNumMarksToRead()
    17 0x55cf827f728d in DB::MergeTreeData::canUseParallelReplicasBasedOnPKAnalysis()
    18 0x55cf827f627e in DB::MergeTreeData::getQueryProcessingStage()
    19 0x55cf7f2f4969 in DB::InterpreterSelectQuery::getSampleBlockImpl() build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:937:31
    20 0x55cf7f2daa00 in DB::InterpreterSelectQuery::InterpreterSelectQuery()
    24 0x55cf7f520b98 in DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter()
    25 0x55cf7f51b6cd in DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery()
    27 0x55cf7f1d4ea9 in DB::InterpreterFactory::get(std::__1::shared_ptr<DB::IAST>&, std::__1::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) build_docker/./src/Interpreters/InterpreterFactory.cpp:162:16
    28 0x55cf8012e485 in DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) build_docker/./src/Interpreters/executeQuery.cpp:1032:31
    29 0x55cf80121bc1 in DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) build_docker/./src/Interpreters/executeQuery.cpp:1229:30
    30 0x55cf8389295f in DB::TCPHandler::runImpl() build_docker/./src/Server/TCPHandler.cpp:424:24
    31 0x55cf838d7dfb in DB::TCPHandler::run() build_docker/./src/Server/TCPHandler.cpp:2050:9
```

</details>

CI: https://s3.amazonaws.com/clickhouse-test-reports/53214/d99b10c340909ed4ee2e6edf0921e8a2f8561b0d/fuzzer_astfuzzermsan/report.html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-08-18 18:02:46 +02:00
Alexander Tokmakov
0bd378afd9
Merge pull request #53381 from bakwc/bakwc-patch-2
Fixed system.data_skipping_indices for MaterializedMySQL
2023-08-18 14:22:40 +03:00
Alexander Tokmakov
59ecbd8cb1
Merge pull request #53539 from ClickHouse/pufit/fix-keeper-path-check
Fix keeper default path check
2023-08-18 13:53:56 +03:00
Alexander Tokmakov
45157769a5
Merge pull request #53534 from ClickHouse/tavplubix-patch-8
Update test.py
2023-08-18 13:42:33 +03:00
Alexey Milovidov
e7d0edfce6
Merge pull request #53261 from bharatnc/ncb/truncate-db
support TRUNCATE DATABASE
2023-08-18 08:22:32 +03:00
Alexey Milovidov
5fcd775656
Merge pull request #53528 from ClickHouse/fix_the_ci
Make sending logs to the cloud less fragile (and fix an unrelated flaky test)
2023-08-18 08:17:33 +03:00
Alexey Milovidov
3bcc4884d7
Merge pull request #53533 from ClickHouse/cpy
Use long timeout for S3 copy requests
2023-08-18 08:13:32 +03:00
Alexey Milovidov
ff7eeb979b
Merge pull request #53535 from nickitat/fix_test_grpc_protocol
Fix `AddressSanitizer failed to allocate 0x0 (0) bytes of SetAlternateSignalStack` in integration tests
2023-08-18 08:12:07 +03:00
Alexey Milovidov
a882bf1bb8
Merge pull request #53495 from ClickHouse/i_hate_materialized_views
Fix a race condition between RESTART REPLICAS and DROP DATABASE
2023-08-18 06:40:04 +03:00
pufit
2e69a1d3a2 Fix keeper default path check 2023-08-17 22:07:39 -04:00
Anton Popov
b9df41d5e3
Merge pull request #53129 from CurtizJ/non-deterministic-mutations
Allow to execute constant non-deterministic functions in mutations on initiator
2023-08-18 00:26:20 +02:00
robot-ch-test-poll
ce58b90ea1
Merge pull request #53455 from rschu1ze/respect-changed-cgroups-limit
Respect cgroup memory limit when reloading configuration
2023-08-18 00:03:35 +02:00
Nikita Taranov
abcf01b809 impl 2023-08-17 23:51:58 +02:00
Alexander Tokmakov
9c9941ca17
Update test.py 2023-08-17 22:58:23 +03:00
Robert Schulze
cd4c53030e
Merge pull request #53531 from rschu1ze/docs-dict
Clean up messy headers in dictionary docs
2023-08-17 21:43:05 +02:00
Michael Kolupaev
557bfea4d8 Use long timeout for S3 copy requests 2023-08-17 19:36:45 +00:00
robot-clickhouse
81af60eeea
Merge pull request #53440 from ZhiguoZh/20230815-date-arrayjoin-fix
Fix an unexpected behavior in #53152
2023-08-17 20:56:01 +02:00
Robert Schulze
f20dd27ba6
Clean header mess up 2023-08-17 18:47:11 +00:00
pufit
a52249872e
Merge pull request #53313 from jorisgio/joris/max_threads_for_indexes
add max_threads_for_indexes settings
2023-08-17 13:59:53 -04:00
Ilya Yatsishin
a32a1b3237
Merge pull request #53527 from ClickHouse/qoega-patch-5 2023-08-17 18:26:42 +02:00
Alexander Tokmakov
89c365803f
Merge pull request #53424 from ClickHouse/exception_message_patterns6
Less exceptions with runtime format string
2023-08-17 17:58:04 +03:00
Alexander Tokmakov
fdca288a73 make sending logs less fragile 2023-08-17 16:49:50 +02:00
Filipp Ozinov
ce87451b66
Merge branch 'ClickHouse:master' into bakwc-patch-2 2023-08-17 18:37:00 +04:00
Anton Popov
1805dec6ff Merge remote-tracking branch 'upstream/master' into HEAD 2023-08-17 14:09:43 +00:00
Ilya Yatsishin
7115110dad
Update merge-tree-settings.md
parts_to_throw_insert=3000
2023-08-17 15:59:22 +02:00
Alexander Tokmakov
72b52250ea
Merge pull request #53501 from azat/dist-flush_on_detach
Add ability to turn off flush of Distributed on DETACH/DROP/server shutdown
2023-08-17 16:56:04 +03:00
Robert Schulze
ec7daed52f
Merge pull request #50276 from ClibMouse/feature/mergetree-checksum-big-endian-support
MergeTree & SipHash checksum big-endian support
2023-08-17 13:27:18 +02:00
Nikolai Kochetov
8dbe82c192
Merge pull request #53489 from ClickHouse/fix-wrong-column-order-in-parallel-final
Fix wrong columns order for queries with parallel FINAL.
2023-08-17 13:22:13 +02:00
Alexander Tokmakov
f83d0dabea
Merge pull request #53295 from ClickHouse/less_no_parallel
Remove no-parallel tag from some tests
2023-08-17 14:08:35 +03:00
Mikhail f. Shiryaev
b7866ce75f
Merge pull request #53493 from ClickHouse/update-style-checker
Update style checker
2023-08-17 12:56:20 +02:00
Alexander Tokmakov
76702a03bb
Update materialized_with_ddl.py (#53494) 2023-08-17 13:08:27 +03:00
Alexander Tokmakov
9e69120bee fix 2023-08-17 12:03:38 +02:00
Filipp Ozinov
5003406ef7
Style 2023-08-17 13:52:21 +04:00
Robert Schulze
5253a7f650
Respect cgroup memory limit when reloading configuration
Cgroups allows to change the amount of memory available to a process
while it runs. The previous logic calculated the amount of available
memory only once at server startup. As a result, memory thresholds set
via cgroups were not picked up when the settings changed. We now always
incorporate the current limits during re-configuraton.

Note 1: getMemoryAmount() opens/reads a file which is potentially
        expensive. Should be fine though since that happens only when
        the server configuration changes.

Note 2: An better approach would be to treat cgroup limit changes as
        another trigger for ClickHouse server re-configuration (which
        currently only happens when the config files change). Shied away
        from that for now because of the case that when the cgroup limit
        is lowered, there is no guarantee that ClickHouse can shrink the
        memory amount accordingly in time (afaik, it does so only lazily
        by denying new allocations). As a result, the OOM killer would
        kill the server. The same will happen with this PR but at a
        lower implementation complexity.
2023-08-17 09:22:13 +00:00
Robert Schulze
1215a23de6
Merge pull request #53422 from rschu1ze/consistency-fixes-for-caches
Refactorings for configuration of in-memory caches
2023-08-17 11:08:32 +02:00
Robert Schulze
4e77984344
Merge pull request #53487 from rschu1ze/curl-head
curl: update to latest master (fixes CVE-2023-32001)
2023-08-17 11:08:04 +02:00
Robert Schulze
067623a4c9
Merge pull request #53444 from rschu1ze/factorize-constants
Minor: Factorize constants in Annoy index
2023-08-17 11:05:56 +02:00
Robert Schulze
d250adf0a5
Merge pull request #53428 from rschu1ze/invalid-async-metrics-update-period
Protect against invalid asynchronous_metrics_update_period_s settings
2023-08-17 11:05:15 +02:00
Filipp Ozinov
f402d80bce
Test for system.data_skipping_indices with MaterializedMySQL 2023-08-17 12:51:41 +04:00
Filipp Ozinov
e609a9f3d1
Test for system.data_skipping_indices with MaterializedMySQL 2023-08-17 12:50:44 +04:00
Mikhail f. Shiryaev
fd3e6c93ee
Produce standard report for cases when AST failed 2023-08-17 09:39:02 +02:00
Azat Khuzhin
17ca2661a1 Add ability to turn off flush of Distributed on DETACH/DROP/server shutdown
Sometimes you can have tons of data there, i.e. few TiBs, and sending
them on server shutdown does not looks sane (maybe there is a bug and
you need to update/restart to fix flushing).

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-08-17 08:58:06 +02:00
Alexey Milovidov
525cc46b0b
Merge pull request #53400 from rschu1ze/follow-up-52695
Follow-up to #52695: Move tests to a more appropriate place
2023-08-17 07:53:34 +03:00
Alexey Milovidov
51876c9a56
Merge pull request #53499 from ClickHouse/simplify-performance-test
Simplify performance test
2023-08-17 07:47:32 +03:00
robot-ch-test-poll3
30dadcfdcc
Merge pull request #53425 from ClickHouse/trash_for_rdb
Add trash for Replicated database
2023-08-17 04:11:40 +02:00
Alexey Milovidov
c3e6f7e9ae Leave only simplifications 2023-08-17 03:59:15 +02:00
Alexey Milovidov
857856b8b6 Leave only simplifications 2023-08-17 03:58:32 +02:00
Alexey Milovidov
64c829807c Merge branch 'master' of github.com:ClickHouse/ClickHouse into export-logs-in-ci-performance 2023-08-17 03:56:27 +02:00
Michael Kolupaev
ff04660972
Merge pull request #53433 from ClickHouse/exp
Fix 'Context has expired' error properly
2023-08-16 18:40:53 -07:00
Zhiguo Zhou
9467addace
Merge branch 'ClickHouse:master' into 20230815-date-arrayjoin-fix 2023-08-17 09:17:57 +08:00