thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-30 03:22:14 +00:00
Code Issues Packages Projects Releases Wiki Activity
122,650 Commits 9,189 Branches 1,782 Tags 2.2 GiB
c78b94bed6
Commit Graph

122650 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Azat Khuzhin
c78b94bed6 Fix possible UB in inverted indexes (experimental feature) 
It is possible to have incorrect types there in case of index contains
functions, add a proper check after value had been changed.

<details>

<summary>MSan report</summary>

```
==182==WARNING: MemorySanitizer: use-of-uninitialized-value
    0 0x55658547c59e in DB::MergeTreeConditionInverted::traverseASTEquals() build_docker/./src/Storages/MergeTree/MergeTreeIndexInverted.cpp:551:26
    1 0x556585475566 in DB::MergeTreeConditionInverted::traverseAtomAST() build_docker/./src/Storages/MergeTree/MergeTreeIndexInverted.cpp:438:21
    2 0x55658547e73e in DB::MergeTreeConditionInverted::MergeTreeConditionInverted()::$_1::operator()(DB::RPNBuilderTreeNode const&, DB::MergeTreeConditionInverted::RPNElement&) const build_docker/./src/Storages/MergeTree/MergeTreeIndexInverted.cpp:228:73
    9 0x556585489e35 in DB::RPNBuilder<DB::MergeTreeConditionInverted::RPNElement>::traverseTree(DB::RPNBuilderTreeNode const&) build_docker/./src/Storages/MergeTree/RPNBuilder.h:252:14
    10 0x556585489921 in DB::RPNBuilder<DB::MergeTreeConditionInverted::RPNElement>::traverseTree(DB::RPNBuilderTreeNode const&) build_docker/./src/Storages/MergeTree/RPNBuilder.h:239:21
    11 0x556585489921 in DB::RPNBuilder<DB::MergeTreeConditionInverted::RPNElement>::traverseTree(DB::RPNBuilderTreeNode const&) build_docker/./src/Storages/MergeTree/RPNBuilder.h:239:21
    12 0x5565854804e6 in DB::RPNBuilder<DB::MergeTreeConditionInverted::RPNElement>::RPNBuilder(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, DB::Block, std::__1::shared_ptr<DB::PreparedSets>, std::__1::function<bool (DB::RPNBuilderTreeNode const&, DB::MergeTreeConditionInverted::RPNElement&)> const&) build_docker/./src/Storages/MergeTree/RPNBuilder.h:218:9
    13 0x55658546fb87 in DB::MergeTreeConditionInverted::MergeTreeConditionInverted(DB::SelectQueryInfo const&, std::__1::shared_ptr<DB::Context const>, DB::Block const&, DB::GinFilterParameters const&, DB::ITokenExtractor const*) build_docker/./src/Storages/MergeTree/MergeTreeIndexInverted.cpp:223:28
    19 0x55658547ceda in DB::MergeTreeIndexInverted::createIndexCondition(DB::SelectQueryInfo const&, std::__1::shared_ptr<DB::Context const>) const build_docker/./src/Storages/MergeTree/MergeTreeIndexInverted.cpp:716:12
    20 0x556587125734 in DB::buildIndexes(std::__1::optional<DB::ReadFromMergeTree::Indexes>&, std::__1::shared_ptr<DB::ActionsDAG>, DB::MergeTreeData const&, std::__1::shared_ptr<DB::Context const> const&, DB::SelectQueryInfo const&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&) build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:1292:48
    21 0x556587121aaa in DB::ReadFromMergeTree::applyFilters() build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:1305:5
    22 0x55658726cf4f in DB::QueryPlanOptimizations::optimizeTreeThirdPass(DB::QueryPlan::Node&, std::__1::list<DB::QueryPlan::Node, std::__1::allocator<DB::QueryPlan::Node>>&) build_docker/./src/Processors/QueryPlan/Optimizations/optimizeTree.cpp:201:38
    23 0x5565870bc489 in DB::QueryPlan::optimize(DB::QueryPlanOptimizationSettings const&) build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:485:5
    24 0x5565870b919a in DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:161:5
    25 0x55658206385a in DB::InterpreterSelectWithUnionQuery::execute() build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:379:31
    26 0x556582c15c50 in DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) build_docker/./src/Interpreters/executeQuery.cpp:751:40
    27 0x556582c09681 in DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) build_docker/./src/Interpreters/executeQuery.cpp:1173:30
    28 0x5565860e7bd7 in DB::TCPHandler::runImpl() build_docker/./src/Server/TCPHandler.cpp:421:24
    29 0x556586126e9e in DB::TCPHandler::run() build_docker/./src/Server/TCPHandler.cpp:2057:9
```

</details>

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-08-18 18:15:17 +02:00
Alexander Tokmakov
0bd378afd9
Merge pull request #53381 from bakwc/bakwc-patch-2 
Fixed system.data_skipping_indices for MaterializedMySQL
 2023-08-18 14:22:40 +03:00
Alexander Tokmakov
59ecbd8cb1
Merge pull request #53539 from ClickHouse/pufit/fix-keeper-path-check 
Fix keeper default path check
 2023-08-18 13:53:56 +03:00
Alexander Tokmakov
45157769a5
Merge pull request #53534 from ClickHouse/tavplubix-patch-8 
Update test.py
 2023-08-18 13:42:33 +03:00
Alexey Milovidov
e7d0edfce6
Merge pull request #53261 from bharatnc/ncb/truncate-db 
support TRUNCATE DATABASE
 2023-08-18 08:22:32 +03:00
Alexey Milovidov
5fcd775656
Merge pull request #53528 from ClickHouse/fix_the_ci 
Make sending logs to the cloud less fragile (and fix an unrelated flaky test)
 2023-08-18 08:17:33 +03:00
Alexey Milovidov
3bcc4884d7
Merge pull request #53533 from ClickHouse/cpy 
Use long timeout for S3 copy requests
 2023-08-18 08:13:32 +03:00
Alexey Milovidov
ff7eeb979b
Merge pull request #53535 from nickitat/fix_test_grpc_protocol 
Fix `AddressSanitizer failed to allocate 0x0 (0) bytes of SetAlternateSignalStack` in integration tests
 2023-08-18 08:12:07 +03:00
Alexey Milovidov
a882bf1bb8
Merge pull request #53495 from ClickHouse/i_hate_materialized_views 
Fix a race condition between RESTART REPLICAS and DROP DATABASE
 2023-08-18 06:40:04 +03:00
pufit
2e69a1d3a2 Fix keeper default path check 2023-08-17 22:07:39 -04:00
Anton Popov
b9df41d5e3
Merge pull request #53129 from CurtizJ/non-deterministic-mutations 
Allow to execute constant non-deterministic functions in mutations on initiator
 2023-08-18 00:26:20 +02:00
robot-ch-test-poll
ce58b90ea1
Merge pull request #53455 from rschu1ze/respect-changed-cgroups-limit 
Respect cgroup memory limit when reloading configuration
 2023-08-18 00:03:35 +02:00
Nikita Taranov
abcf01b809 impl 2023-08-17 23:51:58 +02:00
Alexander Tokmakov
9c9941ca17
Update test.py 2023-08-17 22:58:23 +03:00
Robert Schulze
cd4c53030e
Merge pull request #53531 from rschu1ze/docs-dict 
Clean up messy headers in dictionary docs
 2023-08-17 21:43:05 +02:00
Michael Kolupaev
557bfea4d8 Use long timeout for S3 copy requests 2023-08-17 19:36:45 +00:00
robot-clickhouse
81af60eeea
Merge pull request #53440 from ZhiguoZh/20230815-date-arrayjoin-fix 
Fix an unexpected behavior in #53152
 2023-08-17 20:56:01 +02:00
Robert Schulze
f20dd27ba6
Clean header mess up 2023-08-17 18:47:11 +00:00
pufit
a52249872e
Merge pull request #53313 from jorisgio/joris/max_threads_for_indexes 
add max_threads_for_indexes settings
 2023-08-17 13:59:53 -04:00
Ilya Yatsishin
a32a1b3237
Merge pull request #53527 from ClickHouse/qoega-patch-5 2023-08-17 18:26:42 +02:00
Alexander Tokmakov
89c365803f
Merge pull request #53424 from ClickHouse/exception_message_patterns6 
Less exceptions with runtime format string
 2023-08-17 17:58:04 +03:00
Alexander Tokmakov
fdca288a73 make sending logs less fragile 2023-08-17 16:49:50 +02:00
Filipp Ozinov
ce87451b66
Merge branch 'ClickHouse:master' into bakwc-patch-2 2023-08-17 18:37:00 +04:00
Anton Popov
1805dec6ff Merge remote-tracking branch 'upstream/master' into HEAD 2023-08-17 14:09:43 +00:00
Ilya Yatsishin
7115110dad
Update merge-tree-settings.md 
parts_to_throw_insert=3000
 2023-08-17 15:59:22 +02:00
Alexander Tokmakov
72b52250ea
Merge pull request #53501 from azat/dist-flush_on_detach 
Add ability to turn off flush of Distributed on DETACH/DROP/server shutdown
 2023-08-17 16:56:04 +03:00
Robert Schulze
ec7daed52f
Merge pull request #50276 from ClibMouse/feature/mergetree-checksum-big-endian-support 
MergeTree & SipHash checksum big-endian support
 2023-08-17 13:27:18 +02:00
Nikolai Kochetov
8dbe82c192
Merge pull request #53489 from ClickHouse/fix-wrong-column-order-in-parallel-final 
Fix wrong columns order for queries with parallel FINAL.
 2023-08-17 13:22:13 +02:00
Alexander Tokmakov
f83d0dabea
Merge pull request #53295 from ClickHouse/less_no_parallel 
Remove no-parallel tag from some tests
 2023-08-17 14:08:35 +03:00
Mikhail f. Shiryaev
b7866ce75f
Merge pull request #53493 from ClickHouse/update-style-checker 
Update style checker
 2023-08-17 12:56:20 +02:00
Alexander Tokmakov
76702a03bb
Update materialized_with_ddl.py (#53494) 2023-08-17 13:08:27 +03:00
Alexander Tokmakov
9e69120bee fix 2023-08-17 12:03:38 +02:00
Filipp Ozinov
5003406ef7
Style 2023-08-17 13:52:21 +04:00
Robert Schulze
5253a7f650
Respect cgroup memory limit when reloading configuration 
Cgroups allows to change the amount of memory available to a process
while it runs. The previous logic calculated the amount of available
memory only once at server startup. As a result, memory thresholds set
via cgroups were not picked up when the settings changed. We now always
incorporate the current limits during re-configuraton.

Note 1: getMemoryAmount() opens/reads a file which is potentially
        expensive. Should be fine though since that happens only when
        the server configuration changes.

Note 2: An better approach would be to treat cgroup limit changes as
        another trigger for ClickHouse server re-configuration (which
        currently only happens when the config files change). Shied away
        from that for now because of the case that when the cgroup limit
        is lowered, there is no guarantee that ClickHouse can shrink the
        memory amount accordingly in time (afaik, it does so only lazily
        by denying new allocations). As a result, the OOM killer would
        kill the server. The same will happen with this PR but at a
        lower implementation complexity.
 2023-08-17 09:22:13 +00:00
Robert Schulze
1215a23de6
Merge pull request #53422 from rschu1ze/consistency-fixes-for-caches 
Refactorings for configuration of in-memory caches
 2023-08-17 11:08:32 +02:00
Robert Schulze
4e77984344
Merge pull request #53487 from rschu1ze/curl-head 
curl: update to latest master (fixes CVE-2023-32001)
 2023-08-17 11:08:04 +02:00
Robert Schulze
067623a4c9
Merge pull request #53444 from rschu1ze/factorize-constants 
Minor: Factorize constants in Annoy index
 2023-08-17 11:05:56 +02:00
Robert Schulze
d250adf0a5
Merge pull request #53428 from rschu1ze/invalid-async-metrics-update-period 
Protect against invalid asynchronous_metrics_update_period_s settings
 2023-08-17 11:05:15 +02:00
Filipp Ozinov
f402d80bce
Test for system.data_skipping_indices with MaterializedMySQL 2023-08-17 12:51:41 +04:00
Filipp Ozinov
e609a9f3d1
Test for system.data_skipping_indices with MaterializedMySQL 2023-08-17 12:50:44 +04:00
Mikhail f. Shiryaev
fd3e6c93ee
Produce standard report for cases when AST failed 2023-08-17 09:39:02 +02:00
Azat Khuzhin
17ca2661a1 Add ability to turn off flush of Distributed on DETACH/DROP/server shutdown 
Sometimes you can have tons of data there, i.e. few TiBs, and sending
them on server shutdown does not looks sane (maybe there is a bug and
you need to update/restart to fix flushing).

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-08-17 08:58:06 +02:00
Alexey Milovidov
525cc46b0b
Merge pull request #53400 from rschu1ze/follow-up-52695 
Follow-up to #52695: Move tests to a more appropriate place
 2023-08-17 07:53:34 +03:00
Alexey Milovidov
51876c9a56
Merge pull request #53499 from ClickHouse/simplify-performance-test 
Simplify performance test
 2023-08-17 07:47:32 +03:00
robot-ch-test-poll3
30dadcfdcc
Merge pull request #53425 from ClickHouse/trash_for_rdb 
Add trash for Replicated database
 2023-08-17 04:11:40 +02:00
Alexey Milovidov
c3e6f7e9ae Leave only simplifications 2023-08-17 03:59:15 +02:00
Alexey Milovidov
857856b8b6 Leave only simplifications 2023-08-17 03:58:32 +02:00
Alexey Milovidov
64c829807c Merge branch 'master' of github.com:ClickHouse/ClickHouse into export-logs-in-ci-performance 2023-08-17 03:56:27 +02:00
Michael Kolupaev
ff04660972
Merge pull request #53433 from ClickHouse/exp 
Fix 'Context has expired' error properly
 2023-08-16 18:40:53 -07:00
Zhiguo Zhou
9467addace
Merge branch 'ClickHouse:master' into 20230815-date-arrayjoin-fix 2023-08-17 09:17:57 +08:00