Commit Graph

92277 Commits

Author SHA1 Message Date
FArthur-cmd
21df8ed12f refactoring 2022-07-04 17:21:50 +03:00
FArthur-cmd
21b99e7304 Merge branch 'master' of https://github.com/Vector-Similarity-Search-for-ClickHouse/ClickHouse into annoy-2 2022-07-04 14:54:21 +03:00
Alexander Tokmakov
225c47a457
Merge pull request #38798 from ClickHouse/tavplubix-patch-1
Temporarily disable 01710_projection_fetch_long in BC check
2022-07-04 14:24:56 +03:00
Alexander Tokmakov
a7604a19e9
Update 01710_projection_fetch_long.sql 2022-07-04 14:23:30 +03:00
Ilya Yatsishin
703c3f1938
Merge pull request #38582 from qoega/integration-postgres-better 2022-07-04 13:00:54 +02:00
Ilya Yatsishin
0e4c65a378
Update tests/integration/test_postgresql_database_engine/test.py
Co-authored-by: alesapin <alesapin@gmail.com>
2022-07-04 12:57:55 +02:00
Mikhail f. Shiryaev
1018eff5b3
Merge pull request #38273 from ClickHouse/38218-oo-space
Improve runners AMI and init scripts
2022-07-04 12:39:58 +02:00
Mikhail f. Shiryaev
7c812df47e
Merge pull request #38580 from ClickHouse/artifactory-arm64
Push arm64 packages to artifactory and release assets
2022-07-04 12:31:52 +02:00
Alexander Tokmakov
e49f9cb655
Merge pull request #38749 from azat/tests-cleanup
Tiny tests cleanup
2022-07-04 13:15:15 +03:00
Mikhail f. Shiryaev
7a58a8c19b
Merge pull request #38647 from ClickHouse/clickhouse-diagnostics
Clickhouse diagnostics
2022-07-04 12:14:33 +02:00
alesapin
7378468b4a
Merge pull request #38752 from Algunenano/rmarin_rep_fix
Fix replication after improper merge process
2022-07-04 11:06:04 +02:00
alesapin
be9015852a
Merge pull request #38656 from ClickHouse/some_changes_in_keeper
Add test for keeper `mntr` command
2022-07-04 11:00:18 +02:00
Mikhail f. Shiryaev
446ead1af4
Fix typos issues it clickhouse-diagnostics 2022-07-04 10:59:39 +02:00
Mikhail f. Shiryaev
664d43e348
Add disabling tag no_docker to some go tests 2022-07-04 10:59:39 +02:00
Mikhail f. Shiryaev
7388b6fb55
Test and build clickhouse-diagnostics in release builds 2022-07-04 10:59:37 +02:00
Alexander Gololobov
612e836e60
Merge pull request #38740 from ClickHouse/array_norm_vectorize
Improved vectorized execution of main loop for array norm/distance
2022-07-04 10:19:57 +02:00
Mikhail f. Shiryaev
3f1ab2af8f
Merge pull request #38788 from ClickHouse/revert-38653-deployment-self-extracting
Revert "Upload to S3 compressed self-extracting clickhouse"
2022-07-04 10:15:40 +02:00
Mikhail f. Shiryaev
19516c768f
Revert "Upload to S3 compressed self-extracting clickhouse" 2022-07-04 10:15:27 +02:00
Vitaly Baranov
b4103c1a0e
Merge pull request #38674 from vitlibar/fix-crash-when-grant-all-on-cluster
Fix crash when granting ALL on cluster.
2022-07-04 10:13:56 +02:00
Yakov Olkhovskiy
c415543334
Merge pull request #38653 from ClickHouse/deployment-self-extracting
Upload to S3 compressed self-extracting clickhouse
2022-07-04 01:45:09 -04:00
Alexey Milovidov
c711012399
Merge pull request #38731 from azat/views-max_insert_threads
Fix number of threads for pushing to views
2022-07-04 07:43:26 +03:00
Alexey Milovidov
7f216d1b99
Merge pull request #38773 from kssenii/fix-s3-seekable-read-exception
Fix throwing exception on error for seekable read from s3
2022-07-04 05:41:58 +03:00
kssenii
a772a09149 Fix 2022-07-04 00:14:45 +02:00
Igor Nikonov
2e2ef08712
Merge pull request #37803 from ClickHouse/dictinct_in_order_optimization
DISTINCT in order optimization
2022-07-03 21:59:04 +02:00
Alexey Milovidov
bfc9ed6172
Merge pull request #38722 from ClickHouse/fix-rank-corr-functions
Fix a bug in `rankCorr` function
2022-07-03 20:30:32 +03:00
Alexey Milovidov
1ee752b9a5
Merge pull request #38748 from azat/Map-combinator-fix
Fix use-after-free for Map combinator that leads to incorrect result
2022-07-03 19:32:17 +03:00
Igor Nikonov
2577b59f4c
Merge pull request #38721 from ClickHouse/sort_description_4_read_from_merge_tree
Provide sort description for output stream in ReadFromMergeTree step
2022-07-03 18:06:25 +02:00
Raúl Marín
838fd1e41e Fix replication after improper merge process 2022-07-03 15:11:25 +02:00
Azat Khuzhin
d863f6ce1e tests: add no-backward-compatibility-check for 02351_Map_combinator_dist
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-07-03 15:20:05 +03:00
Azat Khuzhin
7427adb600 tests: disable 01275_parallel_mv under S3 (since it has thread pool for writes)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-07-03 15:14:05 +03:00
Azat Khuzhin
9225256dea tests: improve 01275_parallel_mv
- cover optimize_trivial_insert_select=0/1
- cover max_insert_threads
- convert parallel_view_processing into jinja2

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-07-03 15:14:05 +03:00
Azat Khuzhin
dd3515da98 Fix parallel_view_processing with optimize_trivial_insert_select=1 2022-07-03 15:14:05 +03:00
Azat Khuzhin
4ae7db8369 Fix max_insert_threads while pushing to views
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-07-03 15:14:05 +03:00
Azat Khuzhin
8e9c1eaa73 tests/stress: redirect gdb output on stop hang to gdb.log
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-07-03 14:46:17 +03:00
Azat Khuzhin
f1eec0b6e8 tests: remove unused stress script
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-07-03 14:24:35 +03:00
Azat Khuzhin
769017e1f5 tests: move random_str() helper into shell_config.sh
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-07-03 14:24:24 +03:00
Azat Khuzhin
4375a336fd Fix use-after-free for Map combinator that leads to incorrect result
This use-after-free can be reproduced with distributed queries.

Also note, that this is not sumMappedArray() and friends (that
previously called sumMap()) but Map combinator.

You will find ASan report in details.

<details>

    READ of size 8 at 0x62d00012d218 thread T186 (QueryPipelineEx)
    2022.07.03 05:09:40.000234 [ 31956 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 1.23 GiB, peak 1.23 GiB, will set to 1.25 GiB (RSS), difference: 19.51 MiB
    2022.07.03 05:09:41.000137 [ 31956 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 1.25 GiB, peak 1.25 GiB, will set to 1.26 GiB (RSS), difference: 3.76 MiB
        #0 0x1233a0d8 in DB::AggregateFunctionSumData<>::get() const build_docker/../src/AggregateFunctions/AggregateFunctionSum.h:245:16
        #1 0x1233a0d8 in DB::AggregateFunctionSum<>::insertResultInto(char*, DB::IColumn&, DB::Arena*) const build_docker/../src/AggregateFunctions/AggregateFunctionSum.h:536:70
        #2 0x1470f910 in DB::AggregateFunctionMap<char8_t>::insertResultInto() const build_docker/../src/AggregateFunctions/AggregateFunctionMap.h:236:26
        #3 0x147110ce in DB::IAggregateFunctionHelper<>::insertResultIntoBatch() const build_docker/../src/AggregateFunctions/IAggregateFunction.h:618:53
        #4 0x2c4269d7 in void DB::Aggregator::convertToBlockImplFinal<>() const build_docker/../src/Interpreters/Aggregator.cpp:1878:49
        #5 0x2c403b9f in void DB::Aggregator::convertToBlockImpl<>() const build_docker/../src/Interpreters/Aggregator.cpp:1714:13
        #6 0x2be09b53 in DB::Aggregator::prepareBlockAndFillSingleLevel() const::$_2::operator()() const build_docker/../src/Interpreters/Aggregator.cpp:2144:9
        #7 0x2be09b53 in DB::Block DB::Aggregator::prepareBlockAndFill<>() const build_docker/../src/Interpreters/Aggregator.cpp:2000:5
        #8 0x2be09b53 in DB::Aggregator::prepareBlockAndFillSingleLevel() const build_docker/../src/Interpreters/Aggregator.cpp:2150:12
        #9 0x2be37de3 in DB::Aggregator::mergeBlocks() build_docker/../src/Interpreters/Aggregator.cpp:3032:17
        #10 0x308c27f8 in DB::MergingAggregatedBucketTransform::transform() build_docker/../src/Processors/Transforms/MergingAggregatedMemoryEfficientTransform.cpp:360:37

    0x62d00012d218 is located 3608 bytes inside of 32768-byte region [0x62d00012c400,0x62d000134400)
    freed by thread T186 (QueryPipelineEx) here:
        #0 0xd701312 in free (/work1/azat/tmp/upstream/clickhouse-asan+0xd701312) (BuildId: b7977aef37e9f720)
        ...
        #8 0x2e3c22eb in DB::ColumnAggregateFunction::~ColumnAggregateFunction() build_docker/../src/Columns/ColumnAggregateFunction.cpp:89:1
        ...
        #18 0xd9fcdd4 in std::__1::vector<DB::ColumnWithTypeAndName, std::__1::allocator<DB::ColumnWithTypeAndName> >::~vector() build_docker/../contrib/libcxx/include/vector:401:9
        #19 0x2be373f4 in DB::Aggregator::mergeBlocks() build_docker/../contrib/libcxx/include/__memory/unique_ptr.h
        #20 0x308c27f8 in DB::MergingAggregatedBucketTransform::transform() build_docker/../src/Processors/Transforms/MergingAggregatedMemoryEfficientTransform.cpp:360:37

    previously allocated by thread T186 (QueryPipelineEx) here:
        #0 0xd7015be in malloc (/work1/azat/tmp/upstream/clickhouse-asan+0xd7015be) (BuildId: b7977aef37e9f720)
        #1 0xd85190a in Allocator<false, false>::allocNoTrack(unsigned long, unsigned long) build_docker/../src/Common/Allocator.h:227:27
        #2 0xd988d45 in Allocator<false, false>::alloc(unsigned long, unsigned long) build_docker/../src/Common/Allocator.h:96:16
        #3 0xd988d45 in DB::Arena::MemoryChunk::MemoryChunk(unsigned long, DB::Arena::MemoryChunk*) build_docker/../src/Common/Arena.h:54:64
        #4 0xd98904b in DB::Arena::addMemoryChunk(unsigned long) build_docker/../src/Common/Arena.h:122:20
        #5 0xec9542c in DB::Arena::alignedAlloc(unsigned long, unsigned long) build_docker/../src/Common/Arena.h:171:13
        #6 0x1470f123 in DB::AggregateFunctionMap<char8_t>::deserialize() const build_docker/../src/AggregateFunctions/AggregateFunctionMap.h:205:35

</details>

P.S. Thanks to @den-crane for the reproducer.

Fixes: #35359 (cc @den-crane @dongxiao-yang)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-07-03 12:36:30 +03:00
Vitaly Baranov
1a71e44b28
Merge pull request #38024 from nvartolomei/nv/error-if-profile-does-not-exist
Throw exception when xml user profile does not exist
2022-07-03 11:26:08 +02:00
mergify[bot]
50c105a6db
Merge branch 'master' into fix-crash-when-grant-all-on-cluster 2022-07-03 09:24:52 +00:00
Alexander Gololobov
8ce8158f7f Do computations in Float32 (not Float64) for arrays of Float32 2022-07-03 10:33:11 +02:00
Alexander Gololobov
ca2829188d Perf test for norm/distance with long arrays of floats 2022-07-03 08:01:49 +02:00
Dan Roscigno
6384fe23c3
Merge pull request #38743 from ClickHouse/DanRoscigno-patch-2
typo
2022-07-02 19:44:41 -04:00
Dan Roscigno
29ffee8464
typo 2022-07-02 19:24:26 -04:00
Alexey Milovidov
cd8b701e83
Merge pull request #38720 from kitaisreal/sql-create-drop-index-formatting
SQL create drop index fix formatting
2022-07-03 00:38:06 +03:00
Alexander Gololobov
c6691cc5f2 Improved vectorized execution of main loop for array norm/distance 2022-07-02 22:45:22 +02:00
Alexander Gololobov
92cbc2a3b5 Tests with long arrays 2022-07-02 22:45:22 +02:00
Alexey Milovidov
7adc87a37d
Merge pull request #38413 from nickitat/cache_values_inside_hash_table_grower
Precalculate values inside `HashTableGrower`
2022-07-02 23:30:24 +03:00
Alexey Milovidov
c4718049c1
Merge pull request #38705 from nickitat/fix_filling_transform
Fix FillingTransform
2022-07-02 22:33:44 +03:00
Alexey Milovidov
14eed4963f
Merge pull request #38733 from azat/fix-tests-tmp
Fix CLICKHOUSE_TMP in tests (fixes broken CI)
2022-07-02 22:03:37 +03:00
Azat Khuzhin
9b8fc6a5a6 Fix CLICKHOUSE_TMP in tests
Previous it was not updated, and always uses the first CLICKHOUSE_TMP
for all tests (that was run from one thread).

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-07-02 21:47:29 +03:00