Commit Graph

92173 Commits

Author SHA1 Message Date
Azat Khuzhin
cd2d053ce5 Update poco to fix GCS via proxy tunnel usage
GCS server does not handle requests with port, and simply report an
error:

```xml
    <?xml version="1.0"?>
    <?xml version='1.0' encoding='UTF-8'?>
    <Error>
        <Code>InvalidURI</Code>
        <Message>Couldn't parse the specified URI.</Message>
        <Details>Invalid URL: storage.googleapis.com:443/...</Details>
    </Error>
```

Removing the port fixes the issue. Note that there is port in the Host
header anyway.

Note, this is a problem only for proxy in a tunnel mode, since only it
sends such requests, other sends requests directly via HTTP methods.

Refs: https://github.com/ClickHouse/poco/pull/22#22 (cc @Jokser)
Refs: https://github.com/ClickHouse/poco/pull/63
Refs: #38069 (cc @CurtizJ)
Cc: @alesapin @kssenii

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-07-04 08:46:10 +03:00
Alexey Milovidov
c711012399
Merge pull request #38731 from azat/views-max_insert_threads
Fix number of threads for pushing to views
2022-07-04 07:43:26 +03:00
Alexey Milovidov
7f216d1b99
Merge pull request #38773 from kssenii/fix-s3-seekable-read-exception
Fix throwing exception on error for seekable read from s3
2022-07-04 05:41:58 +03:00
kssenii
a772a09149 Fix 2022-07-04 00:14:45 +02:00
Igor Nikonov
2e2ef08712
Merge pull request #37803 from ClickHouse/dictinct_in_order_optimization
DISTINCT in order optimization
2022-07-03 21:59:04 +02:00
Alexey Milovidov
bfc9ed6172
Merge pull request #38722 from ClickHouse/fix-rank-corr-functions
Fix a bug in `rankCorr` function
2022-07-03 20:30:32 +03:00
Alexey Milovidov
1ee752b9a5
Merge pull request #38748 from azat/Map-combinator-fix
Fix use-after-free for Map combinator that leads to incorrect result
2022-07-03 19:32:17 +03:00
Igor Nikonov
2577b59f4c
Merge pull request #38721 from ClickHouse/sort_description_4_read_from_merge_tree
Provide sort description for output stream in ReadFromMergeTree step
2022-07-03 18:06:25 +02:00
Azat Khuzhin
d863f6ce1e tests: add no-backward-compatibility-check for 02351_Map_combinator_dist
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-07-03 15:20:05 +03:00
Azat Khuzhin
7427adb600 tests: disable 01275_parallel_mv under S3 (since it has thread pool for writes)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-07-03 15:14:05 +03:00
Azat Khuzhin
9225256dea tests: improve 01275_parallel_mv
- cover optimize_trivial_insert_select=0/1
- cover max_insert_threads
- convert parallel_view_processing into jinja2

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-07-03 15:14:05 +03:00
Azat Khuzhin
dd3515da98 Fix parallel_view_processing with optimize_trivial_insert_select=1 2022-07-03 15:14:05 +03:00
Azat Khuzhin
4ae7db8369 Fix max_insert_threads while pushing to views
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-07-03 15:14:05 +03:00
Azat Khuzhin
4375a336fd Fix use-after-free for Map combinator that leads to incorrect result
This use-after-free can be reproduced with distributed queries.

Also note, that this is not sumMappedArray() and friends (that
previously called sumMap()) but Map combinator.

You will find ASan report in details.

<details>

    READ of size 8 at 0x62d00012d218 thread T186 (QueryPipelineEx)
    2022.07.03 05:09:40.000234 [ 31956 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 1.23 GiB, peak 1.23 GiB, will set to 1.25 GiB (RSS), difference: 19.51 MiB
    2022.07.03 05:09:41.000137 [ 31956 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 1.25 GiB, peak 1.25 GiB, will set to 1.26 GiB (RSS), difference: 3.76 MiB
        #0 0x1233a0d8 in DB::AggregateFunctionSumData<>::get() const build_docker/../src/AggregateFunctions/AggregateFunctionSum.h:245:16
        #1 0x1233a0d8 in DB::AggregateFunctionSum<>::insertResultInto(char*, DB::IColumn&, DB::Arena*) const build_docker/../src/AggregateFunctions/AggregateFunctionSum.h:536:70
        #2 0x1470f910 in DB::AggregateFunctionMap<char8_t>::insertResultInto() const build_docker/../src/AggregateFunctions/AggregateFunctionMap.h:236:26
        #3 0x147110ce in DB::IAggregateFunctionHelper<>::insertResultIntoBatch() const build_docker/../src/AggregateFunctions/IAggregateFunction.h:618:53
        #4 0x2c4269d7 in void DB::Aggregator::convertToBlockImplFinal<>() const build_docker/../src/Interpreters/Aggregator.cpp:1878:49
        #5 0x2c403b9f in void DB::Aggregator::convertToBlockImpl<>() const build_docker/../src/Interpreters/Aggregator.cpp:1714:13
        #6 0x2be09b53 in DB::Aggregator::prepareBlockAndFillSingleLevel() const::$_2::operator()() const build_docker/../src/Interpreters/Aggregator.cpp:2144:9
        #7 0x2be09b53 in DB::Block DB::Aggregator::prepareBlockAndFill<>() const build_docker/../src/Interpreters/Aggregator.cpp:2000:5
        #8 0x2be09b53 in DB::Aggregator::prepareBlockAndFillSingleLevel() const build_docker/../src/Interpreters/Aggregator.cpp:2150:12
        #9 0x2be37de3 in DB::Aggregator::mergeBlocks() build_docker/../src/Interpreters/Aggregator.cpp:3032:17
        #10 0x308c27f8 in DB::MergingAggregatedBucketTransform::transform() build_docker/../src/Processors/Transforms/MergingAggregatedMemoryEfficientTransform.cpp:360:37

    0x62d00012d218 is located 3608 bytes inside of 32768-byte region [0x62d00012c400,0x62d000134400)
    freed by thread T186 (QueryPipelineEx) here:
        #0 0xd701312 in free (/work1/azat/tmp/upstream/clickhouse-asan+0xd701312) (BuildId: b7977aef37e9f720)
        ...
        #8 0x2e3c22eb in DB::ColumnAggregateFunction::~ColumnAggregateFunction() build_docker/../src/Columns/ColumnAggregateFunction.cpp:89:1
        ...
        #18 0xd9fcdd4 in std::__1::vector<DB::ColumnWithTypeAndName, std::__1::allocator<DB::ColumnWithTypeAndName> >::~vector() build_docker/../contrib/libcxx/include/vector:401:9
        #19 0x2be373f4 in DB::Aggregator::mergeBlocks() build_docker/../contrib/libcxx/include/__memory/unique_ptr.h
        #20 0x308c27f8 in DB::MergingAggregatedBucketTransform::transform() build_docker/../src/Processors/Transforms/MergingAggregatedMemoryEfficientTransform.cpp:360:37

    previously allocated by thread T186 (QueryPipelineEx) here:
        #0 0xd7015be in malloc (/work1/azat/tmp/upstream/clickhouse-asan+0xd7015be) (BuildId: b7977aef37e9f720)
        #1 0xd85190a in Allocator<false, false>::allocNoTrack(unsigned long, unsigned long) build_docker/../src/Common/Allocator.h:227:27
        #2 0xd988d45 in Allocator<false, false>::alloc(unsigned long, unsigned long) build_docker/../src/Common/Allocator.h:96:16
        #3 0xd988d45 in DB::Arena::MemoryChunk::MemoryChunk(unsigned long, DB::Arena::MemoryChunk*) build_docker/../src/Common/Arena.h:54:64
        #4 0xd98904b in DB::Arena::addMemoryChunk(unsigned long) build_docker/../src/Common/Arena.h:122:20
        #5 0xec9542c in DB::Arena::alignedAlloc(unsigned long, unsigned long) build_docker/../src/Common/Arena.h:171:13
        #6 0x1470f123 in DB::AggregateFunctionMap<char8_t>::deserialize() const build_docker/../src/AggregateFunctions/AggregateFunctionMap.h:205:35

</details>

P.S. Thanks to @den-crane for the reproducer.

Fixes: #35359 (cc @den-crane @dongxiao-yang)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-07-03 12:36:30 +03:00
Vitaly Baranov
1a71e44b28
Merge pull request #38024 from nvartolomei/nv/error-if-profile-does-not-exist
Throw exception when xml user profile does not exist
2022-07-03 11:26:08 +02:00
Dan Roscigno
6384fe23c3
Merge pull request #38743 from ClickHouse/DanRoscigno-patch-2
typo
2022-07-02 19:44:41 -04:00
Dan Roscigno
29ffee8464
typo 2022-07-02 19:24:26 -04:00
Alexey Milovidov
cd8b701e83
Merge pull request #38720 from kitaisreal/sql-create-drop-index-formatting
SQL create drop index fix formatting
2022-07-03 00:38:06 +03:00
Alexey Milovidov
7adc87a37d
Merge pull request #38413 from nickitat/cache_values_inside_hash_table_grower
Precalculate values inside `HashTableGrower`
2022-07-02 23:30:24 +03:00
Alexey Milovidov
c4718049c1
Merge pull request #38705 from nickitat/fix_filling_transform
Fix FillingTransform
2022-07-02 22:33:44 +03:00
Alexey Milovidov
14eed4963f
Merge pull request #38733 from azat/fix-tests-tmp
Fix CLICKHOUSE_TMP in tests (fixes broken CI)
2022-07-02 22:03:37 +03:00
Azat Khuzhin
9b8fc6a5a6 Fix CLICKHOUSE_TMP in tests
Previous it was not updated, and always uses the first CLICKHOUSE_TMP
for all tests (that was run from one thread).

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-07-02 21:47:29 +03:00
Alexander Gololobov
92fdd62a26
Merge pull request #38545 from israelg99/squared_l2
Implement L2 Squared Distance and Norm for array and tuples.
2022-07-02 20:24:54 +02:00
Alexey Milovidov
4ba4329e0d Fix bad test 2022-07-02 19:48:42 +02:00
Alexey Milovidov
4d9e3469d0
Merge pull request #38723 from ClickHouse/hardware-benchmark-example-query
Add example query to hardware benchmark
2022-07-02 20:25:52 +03:00
Alexey Milovidov
5574028617
Merge pull request #38728 from azat/expect-tests-debug
Add exp_internal for expect tests
2022-07-02 20:25:35 +03:00
alesapin
a1c1b7f7b4
Merge pull request #38703 from ClickHouse/fix-backports-diff
Fix backports diff
2022-07-02 18:59:14 +02:00
alesapin
42ea79c266
Merge pull request #38531 from ClickHouse/better_data_part_storage_builder
Use of disk batch operations in MergeTree
2022-07-02 17:24:44 +02:00
Azat Khuzhin
e4b81e6de9 Add exp_internal for expect tests
This is to catch possible issues like 01565_reconnect_after_client_error in [1].

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/38417/e33f236d502fe9f7906392f4ceffaee44e3c4ce4/stateless_tests__thread__actions__[3/3].html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-07-02 17:08:06 +03:00
alesapin
b2db49dbf1
Merge branch 'master' into better_data_part_storage_builder 2022-07-02 15:39:44 +02:00
alesapin
aa823325e9
Merge pull request #38702 from ClickHouse/fix_possible_deadlocks
Fix possible deadlocks with `MergeTreeData::Transaction`
2022-07-02 15:36:59 +02:00
mergify[bot]
b016be264c
Merge branch 'master' into squared_l2 2022-07-02 09:17:28 +00:00
Alexander Tokmakov
42f8e07124
Merge pull request #38680 from ClickHouse/followup_38568
Follow-up to #38568
2022-07-02 11:59:27 +03:00
Alexey Milovidov
9056d80f47 Add example query to hardware benchmark 2022-07-02 03:43:32 +02:00
Alexey Milovidov
668f06b2db Fix error 2022-07-02 03:40:20 +02:00
Alexey Milovidov
16a4b6aa35 Fix a bug in rankCorr function 2022-07-02 03:38:18 +02:00
mergify[bot]
12f5250e86
Merge branch 'master' into dictinct_in_order_optimization 2022-07-01 22:51:35 +00:00
Igor Nikonov
9ef8ff5a31 Addressing review comments 2022-07-01 22:50:00 +00:00
mergify[bot]
a3d3e03b14
Merge branch 'master' into followup_38568 2022-07-01 22:49:55 +00:00
Alexey Milovidov
c2486f3637
Merge pull request #38355 from ClibMouse/add-tlsv1_3-test
Adding TLS V13 Test
2022-07-02 01:32:51 +03:00
Alexey Milovidov
af75c6ee20 Merge pull request #38673 from santrancisco/removebrokenlink
Remove broken client library link
2022-07-02 01:16:41 +03:00
Alexey Milovidov
56875b2a17
Merge pull request #38684 from azat/remove-cmake-doc
[RFC] Remove outdated cmake documentation
2022-07-02 01:11:55 +03:00
Alexey Milovidov
8098945492
Merge pull request #38672 from FArthur-cmd/update_hardware_benchmark
Update hardware benchmark script
2022-07-02 01:01:13 +03:00
Igor Nikonov
1d665e9576 Provide sort description for output stream in ReadFromMergeTree step 2022-07-01 21:28:01 +00:00
Maksim Kita
8177608860 SQL create drop index fix formatting 2022-07-01 22:35:20 +02:00
Yakov Olkhovskiy
776a80bff8
Merge pull request #38136 from bigo-sg/translate_qualified_bug
Bug fixed in TranslateQualifiedNamesMatcher
2022-07-01 14:07:00 -04:00
Alexander Tokmakov
b238ed4093 fix stress tests 2022-07-01 17:19:00 +02:00
alesapin
65956975e2
Update registerDiskHDFS.cpp 2022-07-01 17:18:38 +02:00
Mikhail f. Shiryaev
27bd355525
Use cached image from the release branch for backport PRs 2022-07-01 16:49:27 +02:00
Robert Schulze
2a1ede0f5a
Merge pull request #38589 from ClickHouse/fix-zero-bytes-in-haystack
Fix countSubstrings() & position() on patterns with 0-bytes
2022-07-01 16:15:43 +02:00