thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-04 05:22:17 +00:00
Code Issues Packages Projects Releases Wiki Activity
92,731 Commits 9,182 Branches 1,782 Tags 2.3 GiB
93bf22f585
Commit Graph

28413 Commits

Author SHA1 Message Date
kssenii
0f054774f5 Review fixes 2022-07-04 14:14:15 +02:00
kssenii
cfff7c4c28 Merge master 2022-07-04 14:13:26 +02:00
Robert Schulze
ece61f6da3
Fix davenger's review comments 
https://github.com/ClickHouse/ClickHouse/pull/38434#discussion_r907397214
https://github.com/ClickHouse/ClickHouse/pull/38434#discussion_r907385290
https://github.com/ClickHouse/ClickHouse/pull/38434#discussion_r907406097

(the latter is no longer relevant as the affected places were removed in
the meantime)
 2022-07-04 10:43:21 +00:00
Robert Schulze
d547aa7849
Allow non-const pattern array argument in multi[Fuzzy]Match*() 
Resolves #38046
 2022-07-04 10:43:16 +00:00
xiedeyantu
0a69b76180 add supportParallelWrite() in DiskDecorator 2022-07-04 17:21:07 +08:00
alesapin
7378468b4a
Merge pull request #38752 from Algunenano/rmarin_rep_fix 
Fix replication after improper merge process
 2022-07-04 11:06:04 +02:00
Vitaly Baranov
1dba3d9f9c Make temporary files in better places. 2022-07-04 11:02:58 +02:00
alesapin
be9015852a
Merge pull request #38656 from ClickHouse/some_changes_in_keeper 
Add test for keeper `mntr` command
 2022-07-04 11:00:18 +02:00
Alexander Gololobov
612e836e60
Merge pull request #38740 from ClickHouse/array_norm_vectorize 
Improved vectorized execution of main loop for array norm/distance
 2022-07-04 10:19:57 +02:00
Vitaly Baranov
b4103c1a0e
Merge pull request #38674 from vitlibar/fix-crash-when-grant-all-on-cluster 
Fix crash when granting ALL on cluster.
 2022-07-04 10:13:56 +02:00
lingpeng0314
cca94f01ae add function tryGetKeeperDispatcher for context 
- to make sure other case will get error if dispatcher not initialized
 2022-07-04 13:35:08 +08:00
Alexey Milovidov
c711012399
Merge pull request #38731 from azat/views-max_insert_threads 
Fix number of threads for pushing to views
 2022-07-04 07:43:26 +03:00
lgbo-ustc
932e596689 Merge remote-tracking branch 'ck/master' into storagehive_alter 2022-07-04 09:56:00 +08:00
zvonand
eac84351f6 fix behavior 2022-07-04 01:26:07 +03:00
kssenii
a772a09149 Fix 2022-07-04 00:14:45 +02:00
Igor Nikonov
2e2ef08712
Merge pull request #37803 from ClickHouse/dictinct_in_order_optimization 
DISTINCT in order optimization
 2022-07-03 21:59:04 +02:00
Alexey Milovidov
bfc9ed6172
Merge pull request #38722 from ClickHouse/fix-rank-corr-functions 
Fix a bug in `rankCorr` function
 2022-07-03 20:30:32 +03:00
Alexey Milovidov
1ee752b9a5
Merge pull request #38748 from azat/Map-combinator-fix 
Fix use-after-free for Map combinator that leads to incorrect result
 2022-07-03 19:32:17 +03:00
Igor Nikonov
2577b59f4c
Merge pull request #38721 from ClickHouse/sort_description_4_read_from_merge_tree 
Provide sort description for output stream in ReadFromMergeTree step
 2022-07-03 18:06:25 +02:00
Raúl Marín
838fd1e41e Fix replication after improper merge process 2022-07-03 15:11:25 +02:00
Vitaly Baranov
92e0ee0b6f More detailed error messages. 2022-07-03 14:20:19 +02:00
Azat Khuzhin
dd3515da98 Fix parallel_view_processing with optimize_trivial_insert_select=1 2022-07-03 15:14:05 +03:00
Azat Khuzhin
4ae7db8369 Fix max_insert_threads while pushing to views 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-03 15:14:05 +03:00
Azat Khuzhin
4375a336fd Fix use-after-free for Map combinator that leads to incorrect result 
This use-after-free can be reproduced with distributed queries.

Also note, that this is not sumMappedArray() and friends (that
previously called sumMap()) but Map combinator.

You will find ASan report in details.

<details>

    READ of size 8 at 0x62d00012d218 thread T186 (QueryPipelineEx)
    2022.07.03 05:09:40.000234 [ 31956 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 1.23 GiB, peak 1.23 GiB, will set to 1.25 GiB (RSS), difference: 19.51 MiB
    2022.07.03 05:09:41.000137 [ 31956 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 1.25 GiB, peak 1.25 GiB, will set to 1.26 GiB (RSS), difference: 3.76 MiB
        #0 0x1233a0d8 in DB::AggregateFunctionSumData<>::get() const build_docker/../src/AggregateFunctions/AggregateFunctionSum.h:245:16
        #1 0x1233a0d8 in DB::AggregateFunctionSum<>::insertResultInto(char*, DB::IColumn&, DB::Arena*) const build_docker/../src/AggregateFunctions/AggregateFunctionSum.h:536:70
        #2 0x1470f910 in DB::AggregateFunctionMap<char8_t>::insertResultInto() const build_docker/../src/AggregateFunctions/AggregateFunctionMap.h:236:26
        #3 0x147110ce in DB::IAggregateFunctionHelper<>::insertResultIntoBatch() const build_docker/../src/AggregateFunctions/IAggregateFunction.h:618:53
        #4 0x2c4269d7 in void DB::Aggregator::convertToBlockImplFinal<>() const build_docker/../src/Interpreters/Aggregator.cpp:1878:49
        #5 0x2c403b9f in void DB::Aggregator::convertToBlockImpl<>() const build_docker/../src/Interpreters/Aggregator.cpp:1714:13
        #6 0x2be09b53 in DB::Aggregator::prepareBlockAndFillSingleLevel() const::$_2::operator()() const build_docker/../src/Interpreters/Aggregator.cpp:2144:9
        #7 0x2be09b53 in DB::Block DB::Aggregator::prepareBlockAndFill<>() const build_docker/../src/Interpreters/Aggregator.cpp:2000:5
        #8 0x2be09b53 in DB::Aggregator::prepareBlockAndFillSingleLevel() const build_docker/../src/Interpreters/Aggregator.cpp:2150:12
        #9 0x2be37de3 in DB::Aggregator::mergeBlocks() build_docker/../src/Interpreters/Aggregator.cpp:3032:17
        #10 0x308c27f8 in DB::MergingAggregatedBucketTransform::transform() build_docker/../src/Processors/Transforms/MergingAggregatedMemoryEfficientTransform.cpp:360:37

    0x62d00012d218 is located 3608 bytes inside of 32768-byte region [0x62d00012c400,0x62d000134400)
    freed by thread T186 (QueryPipelineEx) here:
        #0 0xd701312 in free (/work1/azat/tmp/upstream/clickhouse-asan+0xd701312) (BuildId: b7977aef37e9f720)
        ...
        #8 0x2e3c22eb in DB::ColumnAggregateFunction::~ColumnAggregateFunction() build_docker/../src/Columns/ColumnAggregateFunction.cpp:89:1
        ...
        #18 0xd9fcdd4 in std::__1::vector<DB::ColumnWithTypeAndName, std::__1::allocator<DB::ColumnWithTypeAndName> >::~vector() build_docker/../contrib/libcxx/include/vector:401:9
        #19 0x2be373f4 in DB::Aggregator::mergeBlocks() build_docker/../contrib/libcxx/include/__memory/unique_ptr.h
        #20 0x308c27f8 in DB::MergingAggregatedBucketTransform::transform() build_docker/../src/Processors/Transforms/MergingAggregatedMemoryEfficientTransform.cpp:360:37

    previously allocated by thread T186 (QueryPipelineEx) here:
        #0 0xd7015be in malloc (/work1/azat/tmp/upstream/clickhouse-asan+0xd7015be) (BuildId: b7977aef37e9f720)
        #1 0xd85190a in Allocator<false, false>::allocNoTrack(unsigned long, unsigned long) build_docker/../src/Common/Allocator.h:227:27
        #2 0xd988d45 in Allocator<false, false>::alloc(unsigned long, unsigned long) build_docker/../src/Common/Allocator.h:96:16
        #3 0xd988d45 in DB::Arena::MemoryChunk::MemoryChunk(unsigned long, DB::Arena::MemoryChunk*) build_docker/../src/Common/Arena.h:54:64
        #4 0xd98904b in DB::Arena::addMemoryChunk(unsigned long) build_docker/../src/Common/Arena.h:122:20
        #5 0xec9542c in DB::Arena::alignedAlloc(unsigned long, unsigned long) build_docker/../src/Common/Arena.h:171:13
        #6 0x1470f123 in DB::AggregateFunctionMap<char8_t>::deserialize() const build_docker/../src/AggregateFunctions/AggregateFunctionMap.h:205:35

</details>

P.S. Thanks to @den-crane for the reproducer.

Fixes: #35359 (cc @den-crane @dongxiao-yang)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-03 12:36:30 +03:00
Vitaly Baranov
1a71e44b28
Merge pull request #38024 from nvartolomei/nv/error-if-profile-does-not-exist 
Throw exception when xml user profile does not exist
 2022-07-03 11:26:08 +02:00
mergify[bot]
50c105a6db
Merge branch 'master' into fix-crash-when-grant-all-on-cluster 2022-07-03 09:24:52 +00:00
Alexander Gololobov
8ce8158f7f Do computations in Float32 (not Float64) for arrays of Float32 2022-07-03 10:33:11 +02:00
Alexey Milovidov
cd8b701e83
Merge pull request #38720 from kitaisreal/sql-create-drop-index-formatting 
SQL create drop index fix formatting
 2022-07-03 00:38:06 +03:00
Alexander Gololobov
c6691cc5f2 Improved vectorized execution of main loop for array norm/distance 2022-07-02 22:45:22 +02:00
Alexey Milovidov
7adc87a37d
Merge pull request #38413 from nickitat/cache_values_inside_hash_table_grower 
Precalculate values inside `HashTableGrower`
 2022-07-02 23:30:24 +03:00
Alexey Milovidov
c4718049c1
Merge pull request #38705 from nickitat/fix_filling_transform 
Fix FillingTransform
 2022-07-02 22:33:44 +03:00
Alexander Gololobov
92fdd62a26
Merge pull request #38545 from israelg99/squared_l2 
Implement L2 Squared Distance and Norm for array and tuples.
 2022-07-02 20:24:54 +02:00
mergify[bot]
dfac2ca2fc
Merge branch 'master' into fix-crash-when-grant-all-on-cluster 2022-07-02 17:38:45 +00:00
alesapin
dde76824ca Fix ub 2022-07-02 16:02:42 +02:00
alesapin
b2db49dbf1
Merge branch 'master' into better_data_part_storage_builder 2022-07-02 15:39:44 +02:00
alesapin
aa823325e9
Merge pull request #38702 from ClickHouse/fix_possible_deadlocks 
Fix possible deadlocks with `MergeTreeData::Transaction`
 2022-07-02 15:36:59 +02:00
zvonand
8e99ea84a8 fix LOGICAL_ERROR 2022-07-02 14:09:51 +03:00
mergify[bot]
b016be264c
Merge branch 'master' into squared_l2 2022-07-02 09:17:28 +00:00
Azat Khuzhin
e8f5cd3c68 Add separate option to omit symbols from heavy contrib 
Sometimes it is useful to build contrib with debug symbols for further
debugging.

With everything turned ON (i.e. debug build) I got 3.3GB vs 3.0GB w/o
this patch, 9% bloat, thoughts about this is this OK or not for you, if
not STRIP_DEBUG_SYMBOLS_HEAVY_CONTRIB can be OFF by default (regardless
of build type).

P.S. aws debug symbols adds just 1.7%.
v2: rename STRIP_HEAVY_DEBUG_SYMBOLS
v3: OMIT_HEAVY_DEBUG_SYMBOLS
v4: documentation had been removed
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-07-02 06:32:03 +03:00
Alexey Milovidov
668f06b2db Fix error 2022-07-02 03:40:20 +02:00
mergify[bot]
12f5250e86
Merge branch 'master' into dictinct_in_order_optimization 2022-07-01 22:51:35 +00:00
Igor Nikonov
9ef8ff5a31 Addressing review comments 2022-07-01 22:50:00 +00:00
Igor Nikonov
1d665e9576 Provide sort description for output stream in ReadFromMergeTree step 2022-07-01 21:28:01 +00:00
Maksim Kita
8177608860 SQL create drop index fix formatting 2022-07-01 22:35:20 +02:00
Yakov Olkhovskiy
776a80bff8
Merge pull request #38136 from bigo-sg/translate_qualified_bug 
Bug fixed in TranslateQualifiedNamesMatcher
 2022-07-01 14:07:00 -04:00
Kruglov Pavel
eaf8397c13
Fix style 2022-07-01 19:51:07 +02:00
kssenii
831d2253ce Merge master 2022-07-01 18:53:20 +02:00
Amos Bird
53f47127e9
Fix only_merge header 2022-07-01 23:31:45 +08:00
alesapin
65956975e2
Update registerDiskHDFS.cpp 2022-07-01 17:18:38 +02:00
Kruglov Pavel
fe0dad5429
Update src/Common/TimerDescriptor.cpp 2022-07-01 16:23:20 +02:00
Kruglov Pavel
4550acdc35
Update src/Common/Epoll.cpp 
Co-authored-by: Azat Khuzhin <a3at.mail@gmail.com>
 2022-07-01 16:23:03 +02:00
Robert Schulze
2a1ede0f5a
Merge pull request #38589 from ClickHouse/fix-zero-bytes-in-haystack 
Fix countSubstrings() & position() on patterns with 0-bytes
 2022-07-01 16:15:43 +02:00
Nikita Taranov
8ba3d405de impl 2022-07-01 16:05:32 +02:00
alesapin
f47f761cc4
Update src/Disks/ObjectStorages/HDFS/registerDiskHDFS.cpp 
Co-authored-by: Dmitry Novik <n0vik@clickhouse.com>
 2022-07-01 16:04:20 +02:00
zvonand
3b5332d15e Revert "Revert "Non Negative Derivative window function"" 
This reverts commit dea3b5bfce.
 2022-07-01 18:59:07 +05:00
alesapin
919baf331e Review fixes 2022-07-01 15:57:24 +02:00
alesapin
0d7298f3a9 Fix typos 2022-07-01 15:26:27 +02:00
alesapin
57284e6a9d Fix possible deadlocks with MergeTreeData::Transaction 2022-07-01 15:16:32 +02:00
Vitaly Baranov
8195aa768b Move checking if parent profile is allowed to UsersConfigAccessStorage. 2022-07-01 14:46:35 +02:00
Yatsishin Ilya
202008e80b Add asserts 2022-07-01 11:00:04 +00:00
Amos Bird
84a407f381
Fix toHour monotonicity 2022-07-01 18:24:24 +08:00
Vitaly Baranov
ae2f586170 Fix crash when granting ALL on cluster. 2022-07-01 12:19:56 +02:00
alesapin
27a76cb627 Overwrite in encrypted disk 2022-07-01 11:44:51 +02:00
Vitaly Baranov
cadb496894
Merge pull request #38537 from vitlibar/backup-improvements-8 
Backup Improvements 8
 2022-07-01 11:20:46 +02:00
Amos Bird
d73f975849
Fix intermediate header in aggregate projection 2022-07-01 16:47:32 +08:00
mergify[bot]
6e75d47f51
Merge branch 'master' into keeper-dont-rollback-session-id 2022-07-01 07:53:50 +00:00
lingpeng0314
03a5ce542d Remove Exception in getKeeperDispatcher 
- to handle the nil case by caller
 2022-07-01 11:27:40 +08:00
lgbo-ustc
bab954c461 update codes based on review comment 2022-07-01 10:15:35 +08:00
Alexey Milovidov
20841f0e1e
Merge pull request #38551 from ClickHouse/revert-37628-non-neg-deriv 
Revert "Non Negative Derivative window function"
 2022-07-01 02:46:28 +03:00
alesapin
66705eacb0 Add test for keeper mntr command 2022-07-01 00:07:22 +02:00
alesapin
dfb0dfc4a7 Less diff 2022-06-30 23:11:20 +02:00
alesapin
09faa1641e Some comments 2022-06-30 23:09:32 +02:00
alesapin
b7c0d25528 Less code 2022-06-30 23:01:55 +02:00
alesapin
3afd0c4ab6 Remove redundant method 2022-06-30 22:57:39 +02:00
alesapin
979565edf0 Merge branch 'master' into better_data_part_storage_builder 2022-06-30 22:54:42 +02:00
alesapin
eb5046ab26 Simplify everything 2022-06-30 22:51:27 +02:00
Alexander Tokmakov
60dcae9af0
Merge pull request #38627 from Algunenano/rmarin_fix_replicated_part_2 
Adapt some more nodes to avoid issues with pre-22.4 replicas
 2022-06-30 21:11:35 +03:00
Maksim Kita
3a1d8f4518
Merge pull request #38565 from ClickHouse/merging-column-vector-replicate-optimize 
Merging #37235
 2022-06-30 18:29:51 +02:00
Maksim Kita
4e55550490
Merge pull request #38602 from kitaisreal/cache-dictionary-simplify-update-queue 
CacheDictionary simplify update queue
 2022-06-30 18:26:26 +02:00
avogar
5b0fd31c64 Put column names in quotes 2022-06-30 16:14:30 +00:00
Alexander Tokmakov
d2cbdc7c53
Update ReplicatedMergeTreeQueue.cpp 2022-06-30 17:23:53 +03:00
Maksim Kita
a0e5768956
Merge pull request #38623 from loyispa/patch 
Fix typo
 2022-06-30 16:02:36 +02:00
Vitaly Baranov
e367d96964 Fix style. 2022-06-30 15:10:33 +02:00
Raúl Marín
cbcd740dc1 Adapt some more nodes to avoid issues with pre-22.4 replicas 2022-06-30 15:10:09 +02:00
Igor Nikonov
488ee75fc4 + use DistinctSorted for final distinct step 
+ fix performance tests
 2022-06-30 13:03:39 +00:00
Maksim Kita
d31ca4c4b6 Fixed tests 2022-06-30 14:49:11 +02:00
alesapin
1363b478d4 Merge branch 'fix_disk_tx_write_op' into better_data_part_storage_builder 2022-06-30 14:38:32 +02:00
alesapin
845264dd7b Followup 2022-06-30 14:36:18 +02:00
alesapin
dee6240494 Remove redundant 2022-06-30 14:34:32 +02:00
alesapin
c5943d0384 Better comment 2022-06-30 14:33:12 +02:00
alesapin
1ff4222661 Remove redundant changes 2022-06-30 14:33:09 +02:00
alesapin
9a89154352 Fix incorrect order of operations in disk transactions 
(cherry picked from commit 0d3904b788)
 2022-06-30 14:33:03 +02:00
Anton Popov
7c721578c7
Merge pull request #38320 from CurtizJ/dynamic-columns-16 
Improve performace of insertion to columns of type JSON
 2022-06-30 14:18:20 +02:00
alesapin
cb90aca2ef Some changes in merge tree 2022-06-30 14:12:45 +02:00
Alexander Gololobov
b2b31103c5 Reuse common code for L2Squared and L2 2022-06-30 14:12:25 +02:00
alesapin
0d3904b788 Fix incorrect order of operations in disk transactions 2022-06-30 14:11:43 +02:00
Robert Schulze
1e3b5bfcb7
Fix test 00233_position_function_family 2022-06-30 11:43:25 +00:00
Kruglov Pavel
57686b8422
Merge pull request #38561 from kitaisreal/sql-create-drop-index-update 
SQL create drop index minor fixes
 2022-06-30 13:38:39 +02:00
Maksim Kita
95687f2d01 CacheDictionaryUpdateUnit make update state non atomic 2022-06-30 13:15:35 +02:00
Maksim Kita
0de66a2712
Merge pull request #38449 from ClickHouse/revert-38361-revert-38324-fix-partial-sort 
Revert "Revert "Fix optimization in PartialSortingTransform (SIGSEGV and possible incorrect result)""
 2022-06-30 13:02:38 +02:00
Azat Khuzhin
a47355877e
Add revision() function (#38555) 
It can be useful to match versions, since in some tables
(system.trace_log) there is only revision column.

P.S. came to this when was digging into stress reports from CI.
P.P.S. case insensitive by analogy with version().

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-30 12:58:26 +02:00
Maksim Kita
f443cf66f0 CacheDictionary simplify update queue 2022-06-30 12:53:56 +02:00
Maksim Kita
83375465eb
Merge pull request #38601 from kitaisreal/dictionaries-added-tsa-annotations 
Dictionaries added TSA annotations
 2022-06-30 12:51:54 +02:00
loyispa
109c9bcbd5 Fix typo 2022-06-30 18:37:19 +08:00
alesapin
be213c0be7
Merge pull request #38590 from ClickHouse/proper_fix 
Proper fix for ipv4/ipv6 conversion error
 2022-06-30 11:15:01 +02:00
Vitaly Baranov
5456bde4a2 Improve gathering metadata for storing ACL in backups. 2022-06-30 09:46:37 +02:00
mergify[bot]
e3c00ce4f3
Merge branch 'master' into zookeeper-add-extra-list-argument 2022-06-30 07:17:23 +00:00
mergify[bot]
1795f863a5
Merge branch 'master' into keeper-dont-rollback-session-id 2022-06-30 07:16:43 +00:00
Antonio Andelic
de264117fd
Merge pull request #38118 from bigo-sg/storagehive_struct_type 
Add struct type support in `StorageHive`
 2022-06-30 09:11:13 +02:00
Vitaly Baranov
031ca28fdc Add test for partition clause. More checks for data compatibility on restore. 2022-06-30 08:37:18 +02:00
Vitaly Baranov
11b51d2878 Implement storing UDF in backups. 2022-06-30 08:37:17 +02:00
Vitaly Baranov
aa97bf5125 Improve handling predefined databases and tables. 2022-06-30 08:37:17 +02:00
Vitaly Baranov
efbee5e723 Improve gathering metadata for backup - part 7. 2022-06-30 08:37:17 +02:00
Vitaly Baranov
01921ce9a3 Move most code from setTableStructure() to a separate function. 2022-06-30 08:37:17 +02:00
Vitaly Baranov
7689e0c36f Improve gathering metadata for backup - part 6. 2022-06-30 08:37:17 +02:00
Vitaly Baranov
6ca400fd89 Improve gathering metadata for backup - part 5. 2022-06-30 08:37:17 +02:00
Vitaly Baranov
aaf7f66549 Improve gathering metadata for backup - part 4. 2022-06-30 08:37:17 +02:00
Vitaly Baranov
44db346fea Improve gathering metadata for backup - part 3. 2022-06-30 08:37:17 +02:00
Vitaly Baranov
461a31f237 Improve gathering metadata for backup - part 2. 2022-06-30 08:37:17 +02:00
Vitaly Baranov
64b51a3772 Improve gathering metadata for backup. 2022-06-30 08:37:17 +02:00
Vitaly Baranov
47ac47350b Store projections in backups. 2022-06-30 08:37:17 +02:00
mergify[bot]
4cbbfb431d
Merge branch 'master' into dictinct_in_order_optimization 2022-06-29 23:32:17 +00:00
Igor Nikonov
d435532c68 Adapt range search algorithm to high cardinality case 
+ range search done in steps of some number of rows.
  Controled by new
  setting `distinct_in_order_range_search_step`. By default 0, i.e.
  whole chunk
+ before start binary search, linear probing is done on each step (32
  rows currently)
 2022-06-29 23:30:35 +00:00
alesapin
8726c7df26 Strangest fix for HDFS test 2022-06-30 01:21:49 +02:00
Igor Nikonov
bd4a208428
Merge pull request #38438 from ClickHouse/distinct_sorted_small_refact 
Distinct sorted: calculate column positions once
 2022-06-30 00:53:17 +02:00
Robert Schulze
81bb2242fd
Fix countSubstrings() & position() on patterns with 0-bytes 
SQL functions countSubstrings(), countSubstringsCaseInsensitive(),
countSubstringsUTF8(), position(), positionCaseInsensitive(),
positionUTF8() with non-const pattern argument use fallback sorters
LibCASCIICaseSensitiveStringSearcher and LibCASCIICaseInsensitiveStringSearcher
which call ::strstr(), resp. ::strcasestr(). These functions assume that
the haystack is 0-terminated and they even document that. However, the
callers did not check if the haystack contains 0-byte (perhaps because
its sort of expensive). As a consequence, if the haystack contained a
zero byte in it's payload, matches behind this zero byte were ignored.

    create table t (id UInt32, pattern String) engine = MergeTree() order by id;
    insert into t values (1, 'x');
    select countSubstrings('aaaxxxaa\0xxx', pattern) from t;

We returned 3 before this commit, now we return 6
 2022-06-29 21:41:18 +00:00
Nikita Taranov
7d2e8b0883 clean up 2022-06-29 22:15:12 +02:00
Dmitry Novik
4f7967ab9e
Merge pull request #37887 from azat/union-fix 
Fix incorrect columns order in subqueries of UNION
 2022-06-29 20:40:23 +02:00
alesapin
8f5582f95e Review and style fixes 2022-06-29 20:29:50 +02:00
Maksim Kita
09be594c81 Dictionaries added TSA annotations 2022-06-29 19:15:47 +02:00
alesapin
62e7a89f26 Proper fix for ipv4/ipv6 conversion error 2022-06-29 17:53:08 +02:00
Maksim Kita
65110fdffc ColumnVector refactor replicate SSE42 optimization 2022-06-29 17:50:25 +02:00
mergify[bot]
36139eacd7
Merge branch 'master' into dictinct_in_order_optimization 2022-06-29 13:37:16 +00:00
filimonov
e0acb6e337
Add check for empty proccessors in AggregatingTransform::expandPipeline 2022-06-29 15:23:53 +02:00
alesapin
67f2fa0ef0 Fix style 2022-06-29 15:18:48 +02:00
Igor Nikonov
3627c6ff36 Perf tests with high cardinality 2022-06-29 13:13:39 +00:00
alesapin
2f03e821bd Fix stupid bug 2022-06-29 15:08:16 +02:00
Anton Popov
e78814f3bb
Merge pull request #38371 from CurtizJ/fix-distint-with-limit 
Fix `DISTINCT` with `LIMIT` in distributed queries
 2022-06-29 14:02:41 +02:00
alesapin
ac963a7663 Remove redundant check 2022-06-29 13:55:20 +02:00
Maksim Kita
ea0996674f
Merge pull request #37235 from zzachimed/column_replicate_optimize_v2 
Improve performance for column replicate for uint32 type. In our benc…
 2022-06-29 13:45:22 +02:00
Alexander Tokmakov
8246e55002
Merge pull request #38486 from azat/fix-parts-removal 
Fix parts removal after incorrect server shutdown
 2022-06-29 14:17:59 +03:00
Alexander Tokmakov
32c6a944ee
Merge pull request #38541 from Algunenano/rmarin_fix_replicated_create_table_compat 
Fix table creation to avoid replication issues with pre-22.4 replicas
 2022-06-29 14:15:50 +03:00
Maksim Kita
eeae73e0cf SQL create drop index update implementation 2022-06-29 13:10:30 +02:00
mergify[bot]
1280fc39a0
Merge branch 'master' into better-exception-messages 2022-06-29 11:09:06 +00:00
Kruglov Pavel
b9813bcdea
Merge pull request #38356 from kssenii/fix-rabbitmq-with-peekable 
Fix RabbitMQ with formats based on PeekableReadBuffer
 2022-06-29 13:06:10 +02:00
mergify[bot]
9482c99ab8
Merge branch 'master' into sql-insert-format 2022-06-29 11:03:07 +00:00
Kruglov Pavel
77873e75bf
Update TimerDescriptor.cpp 2022-06-29 13:01:14 +02:00
Kruglov Pavel
30fd086b5e
Add logging in Epoll and TimerDescriptor in case of EINTR 2022-06-29 12:59:18 +02:00
Alexander Tokmakov
c3573532c0
Merge pull request #38487 from ClickHouse/add_some_tsa_annotations 
Add some TSA annotations
 2022-06-29 13:57:10 +03:00
Alexander Tokmakov
ceb66ade4b
Merge pull request #38335 from ClickHouse/deprecate_ordinary_database 
Deprecate Ordinary database and old *MergeTree syntax
 2022-06-29 13:42:59 +03:00
alesapin
c80a4c27be Merge branch 'master' into better_data_part_storage_builder 2022-06-29 12:33:42 +02:00
Maksim Kita
23e09b5e9b
Merge pull request #35166 from zhangjmruc/sql_compatible_create_drop_index 
Support SQL compatible create/drop index syntax
 2022-06-29 12:30:28 +02:00
Antonio Andelic
2de659715b Don't rollback SessionID request 2022-06-29 10:21:37 +00:00
mergify[bot]
26258959b1
Merge branch 'master' into distinct_sorted_small_refact 2022-06-29 09:38:34 +00:00
Antonio Andelic
7a8fbbf132 Merge branch 'master' into zookeeper-add-extra-list-argument 2022-06-29 08:31:03 +00:00
Antonio Andelic
1d26446f84 Define new list request type 2022-06-29 08:30:39 +00:00
Robert Schulze
bb441faa42
Merge pull request #38491 from ClickHouse/clang-tidy-in-branches 
Fix some clang-tidy warnings in headers
 2022-06-29 10:22:04 +02:00
Alexey Milovidov
0d27672ac8
Merge pull request #38440 from xiedeyantu/allow_trailing_comma 
Allow trailing comma in columns list
 2022-06-29 09:47:09 +03:00
Alexey Milovidov
e7dbe526f8
Update ParserCreateQuery.cpp 2022-06-29 09:46:45 +03:00
Alexey Milovidov
4cda5491f6
Merge pull request #38542 from ClickHouse/unique_lock-to-lock_guard 
Don't use std::unique_lock unless we need to
 2022-06-29 09:40:56 +03:00
Alexey Milovidov
dea3b5bfce
Revert "Non Negative Derivative window function" 2022-06-29 08:56:15 +03:00
jianmei zhang
d75af09f74 Fix style error 2022-06-29 11:08:05 +08:00
jianmei zhang
8d49aa4697 Support sql standard drop index in seprate parser files 2022-06-29 11:08:05 +08:00
jianmei zhang
589cba8045 Support sql standard create index in seprate parser files 2022-06-29 11:08:05 +08:00
jianmei zhang
9154aec201 Fix compile error 2022-06-29 11:08:05 +08:00
Jianmei Zhang
13231d6a85 Pass command type for create and drop index 2022-06-29 11:08:05 +08:00
Jianmei Zhang
4d0635371f Support sql stanard create and drop index 2022-06-29 11:08:05 +08:00
Nikita Taranov
f5d26572df
Quick fix for aggregation pipeline (#38295) 2022-06-29 01:16:30 +02:00
Igor Nikonov
4a00e33e6b Fixes for some review comments 2022-06-28 21:42:46 +00:00
Igor Nikonov
c1840e798c Fix: wrong header variable was used 2022-06-28 20:15:16 +00:00
Igor Nikonov
d80a21a445 Distinct sorted: calculate column positions once in constructor 
- instead of calculating them on every chunk
 2022-06-28 19:59:05 +00:00
Azat Khuzhin
f86ddebada Add sanity check for UNION header 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-28 22:53:03 +03:00
Azat Khuzhin
d98336ad83 Fix incorrect columns order in subqueries of UNION 
Consider the following query:

    SELECT avgWeighted(x, y) FROM (SELECT NULL, 255 AS x, 1 AS y UNION ALL SELECT y, NULL AS x, 1 AS y)

Here is UNION from two SELECT queries
- `SELECT NULL, 255 AS x, 1 AS y`
- `SELECT y, NULL AS x, 1 AS y`

UNION queries matches columns by positions, not names, so the following
columns should be used by `avgWeighted()`:
- `255 AS x,  1 AS y`
- `NULL AS x, 1 AS y`

Result types of arguments should be:
- `x Nullable(UInt8)`
- `y UInt8`

And in case of UNION query is a subselect itself, it will return only
required columns, for the example above it needs only `x` and `y`.
For this it will get positions of these arguments from the first query,
and then use those positions to get required column names from the
second query (since there is no ability to get columns by positions
instead of names internally), and due to duplicated columns the second
query will return (`y`, `x`) not (`x`, `y`), and this will make the
result incorrect:

    EXPLAIN header = 1, optimize = 0, actions=1 SELECT avgWeighted(x, y) FROM (SELECT NULL, 255 AS x, 1 AS y UNION ALL SELECT y, NULL AS x, 1 AS y)

    Aggregates:
        avgWeighted(x, y)
          Function: avgWeighted(Nullable(UInt8), UInt8) → Nullable(Float64)
          Arguments: x, y
          Argument positions: 0, 1
      Expression (Before GROUP BY)
      Header: x UInt8
              y Nullable(UInt8)
      ...
      Union
      Header: x UInt8
              y Nullable(UInt8)
        Expression (Conversion before UNION)
        Header: x UInt8
                y Nullable(UInt8)
        Expression (Conversion before UNION)
        Header: x UInt8
                y Nullable(UInt8)

And the query itself fails with an error:

    Logical error: 'Bad cast from type DB::ColumnVector<char8_t> to DB::ColumnNullable'.

_NOTE: `avgWeighted()` here is required to trigger `LOGICAL_ERROR`_

CI: https://s3.amazonaws.com/clickhouse-test-reports/37796/e637489f81768df582fe7389e57f7ed12893087c/fuzzer_astfuzzerdebug,actions//report.html
Fixes: 02227_union_match_by_name
v2: fix untuple() (reserve space for output_columns_positions too)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-28 22:53:03 +03:00
Azat Khuzhin
ee0f2651ee Revert "Fix converting types for UNION queries (may produce LOGICAL_ERROR)" 
This fix is incorrect, and it introduce new issues, in particular it
may breaks UNION queries w/o column aliases, i.e.:

    SELECT a, b, c FROM (SELECT 3 AS a, 2147483647 AS b, 1048575 AS c UNION ALL SELECT -2, NULL, -2) AS js1 ORDER BY a

CI: https://s3.amazonaws.com/clickhouse-test-reports/37796/e637489f81768df582fe7389e57f7ed12893087c/fuzzer_astfuzzerdebug,actions//report.html
Reverts: #37593/#34775 (2613149f6b)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-28 22:50:01 +03:00
Julian Gilyadov
d0d72e0b5e
Implement L2Squared Distance and Norm 2022-06-28 15:28:58 -04:00
Robert Schulze
f692ead6ad
Don't use std::unique_lock unless we have to 
Replace where possible by std::lock_guard which is more light-weight.
 2022-06-28 19:19:06 +00:00
Raúl Marín
9b4da86e2e Adapt table creation with replication to avoid issues with pre-22.4 replicas 2022-06-28 20:12:07 +02:00
Alexander Tokmakov
4f54abf67f better names for macros 2022-06-28 16:27:18 +02:00
Anton Popov
58c8facebb minor fixes 2022-06-28 14:21:21 +00:00
mergify[bot]
bed1f68c74
Merge branch 'master' into sql-insert-format 2022-06-28 14:20:31 +00:00
Alexander Tokmakov
021bdc335e Merge branch 'master' into add_some_tsa_annotations 2022-06-28 15:42:01 +02:00
Alexander Tokmakov
428628238f Merge branch 'master' into deprecate_ordinary_database 2022-06-28 15:40:33 +02:00
alesapin
6429b72371 Fixes 2022-06-28 14:41:22 +02:00
Robert Schulze
c22038d48b
More clang-tidy fixes 2022-06-28 11:50:05 +00:00
Miel Donkers
4e9f396a48
Small improvement of the error message to hint at possible issue (#38458) 2022-06-28 13:36:30 +02:00
alesapin
c7620b8217 Use builder instead of disk 2022-06-28 13:19:30 +02:00
kssenii
4c99f608b3 Fix 2022-06-28 13:18:21 +02:00
mergify[bot]
ea416c877f
Merge branch 'master' into better-exception-messages 2022-06-28 11:13:26 +00:00
alesapin
a68bf8fff2 Missed changes 2022-06-28 12:59:54 +02:00
alesapin
0a3fab1cb6 Some sad changes 2022-06-28 12:51:49 +02:00
Alexander Tokmakov
e82a375946
Update Settings.h 2022-06-28 13:47:22 +03:00
Igor Nikonov
64490963fd
Merge pull request #38489 from ClickHouse/cleanup_local_clang_tidy_fixes 
Cleanup: local clang-tidy warnings founded during review
 2022-06-28 12:23:46 +02:00
Kseniia Sumarokova
4a6af1ba81
Merge pull request #38466 from kssenii/follow-uo-iobjectstorage 
Follow up for #38436
 2022-06-28 12:06:42 +02:00
alesapin
49add19842 Merge branch 'better_data_part_storage_builder' of github.com:ClickHouse/ClickHouse into better_data_part_storage_builder 2022-06-28 12:02:27 +02:00
alesapin
1a4aa8e52e Merge branch 'master' into better_data_part_storage_builder 2022-06-28 11:51:06 +02:00
lgbo-ustc
2e3923757c fixed:Moved-from object 'column_array_of_element' is moved 2022-06-28 17:20:33 +08:00
Robert Schulze
3e528c0385
Fix duplicate include 2022-06-28 08:13:51 +00:00
Alexander Tokmakov
09a0688d80 fix 2022-06-28 00:26:52 +02:00
Robert Schulze
5e28580f11
Fix some clang-tidy warnings in headers 2022-06-27 23:49:07 +02:00
Alexander Tokmakov
851534c9f7 Merge branch 'master' into deprecate_ordinary_database 2022-06-27 23:47:49 +02:00
Igor Nikonov
e41d612b1d Cleanup: local clang-tidy warnings founded during review 2022-06-27 20:57:18 +00:00
Azat Khuzhin
e70cff2cea Fix parts removal after incorrect server shutdown 
Before this patch, and after transaction support had been added #24258,
if the server had not removed old parts on shutdown, then they will left
forever.

Fixes: #24258 (cc @tavplubix)
v2: use Tx::PrehistoricTID over removePartsFromWorkingSet() from ctor
v3: apply black to new test
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-27 23:56:25 +03:00
Alexander Tokmakov
f4883f1f7e add some TSA aanotations 2022-06-27 22:48:27 +02:00
Igor Nikonov
59295724ac Mark condition for empty chunk as unlikely 2022-06-27 20:44:39 +00:00
Azat Khuzhin
f0271dc486 Refactor deactive_part() a little 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-27 23:42:39 +03:00
mergify[bot]
a9c1b68034
Merge branch 'master' into dictinct_in_order_optimization 2022-06-27 20:16:00 +00:00
Dmitry Novik
b629d557ce
Merge pull request #38246 from ClickHouse/overcommit-deadlock-fix 
Fix deadlock in OvercommitTracker logging
 2022-06-27 21:57:31 +02:00
Igor Nikonov
5a26349695 Fix: input chunk can have empty columns (no rows) 2022-06-27 19:51:06 +00:00
alesapin
d88cdcd5c1 AAA 2022-06-27 21:41:29 +02:00
Nikita Taranov
986ead7358 use old HashTableGrower with *WithStackMemory data structures 2022-06-27 20:34:18 +02:00
avogar
9bb68bc6de Add SQLInsert output format 2022-06-27 18:31:57 +00:00
Yakov Olkhovskiy
d5f65ece9b
Merge pull request #38105 from arenadata/ADQM-419 
Add kerberosInit function as a replacement for kinit executable calls in Kafka and HDFS
 2022-06-27 14:19:24 -04:00
alesapin
180b4cd64d Merge branch 'master' into better_data_part_storage_builder 2022-06-27 19:46:17 +02:00
Robert Schulze
53cd8820d3
Merge pull request #38434 from ClickHouse/multi-match-with-nonconst-patterns 
Refactoring to enable multi-match functions with non-const needles
 2022-06-27 19:35:48 +02:00
alesapin
bed8ee556c
Merge pull request #38465 from ClickHouse/kssenii-patch-4 
Fix comment
 2022-06-27 18:44:17 +02:00
Alexander Tokmakov
2283436d24
Merge pull request #38344 from Algunenano/implicit_transactions 
Add implicit_transaction setting
 2022-06-27 18:50:37 +03:00
kssenii
c9faab4992 Fix 2022-06-27 16:48:06 +02:00
Kseniia Sumarokova
e515579cbe
Update IObjectStorage.h 2022-06-27 16:26:36 +02:00
Alexander Tokmakov
88dbbcb15d fix tests 2022-06-27 16:10:00 +02:00
avogar
12448f9400 Fix style 2022-06-27 13:56:43 +00:00
avogar
344ddc6e7d Fix tests 2022-06-27 13:55:51 +00:00
Alexander Tokmakov
bec921c9d9
Merge pull request #38265 from ClickHouse/cleanup_garbage_in_store_dir 
Add background cleanup of store/ subdirs
 2022-06-27 16:53:13 +03:00
Kseniia Sumarokova
be283d1c12
Update RabbitMQSource.cpp 2022-06-27 15:39:22 +02:00
Kseniia Sumarokova
cadc202b35
Update RabbitMQSource.cpp 2022-06-27 15:37:58 +02:00
Alexander Tokmakov
7c37079a96 Merge branch 'master' into deprecate_ordinary_database 2022-06-27 14:46:59 +02:00
Anton Kozlov
5f81bcd84f
CLICKHOUSE-1331 Rewrite tuple functions as literals in backwards-compatibility mode (#38096) 
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2022-06-27 14:13:21 +02:00
kssenii
2c5aeaaa1a Add auto close for postgres connection 2022-06-27 13:46:52 +02:00
Nikita Taranov
2487ba7f00
Move updateInputStream to ITransformingStep (#37393) 2022-06-27 13:16:52 +02:00
mergify[bot]
da52780cb5
Merge branch 'master' into implicit_transactions 2022-06-27 09:46:34 +00:00
alesapin
50eb364a56
Merge pull request #38436 from kssenii/remote-path-separation-in-object-storage 
more consistent work with paths in object storages
 2022-06-27 10:58:01 +02:00
Maksim Kita
3ebe6a03b1
Revert "Revert "Fix optimization in PartialSortingTransform (SIGSEGV and possible incorrect result)"" 2022-06-27 10:37:19 +02:00
Antonio Andelic
c06776ce0d Merge branch 'master' into zookeeper-add-extra-list-argument 2022-06-27 06:53:29 +00:00
Antonio Andelic
0d906e311d PR review fixes 2022-06-27 06:53:09 +00:00
xiedeyantu
c768e22e24 update 2022-06-27 13:17:22 +08:00
kssenii
5727671cac Fix tests 2022-06-27 00:34:57 +02:00
Igor Nikonov
edd29707ca Some polishing 2022-06-26 21:44:10 +00:00
alesapin
83b584b2dd Something 2022-06-26 18:43:28 +02:00
Robert Schulze
959cbaab02
Move loop over patterns into implementations 
- This is preparation for non-const regexp arguments, where this loop
  will run for each row.
 2022-06-26 16:26:13 +00:00
Robert Schulze
cb5d1a4a85
Fix style check 2022-06-26 16:25:49 +00:00
Robert Schulze
b8f67185bf
Cosmetics: Whitespaces 2022-06-26 16:25:49 +00:00
Robert Schulze
e2b11899a1
Move check if cfg allows hyperscan into implementations 
- This is not needed for non-const regexp array arguments but cleans up
  the code and runs the check only in functions which actually use
  hyperscan.
 2022-06-26 16:25:49 +00:00
Robert Schulze
c2cea38b97
Move local variable into if statement 2022-06-26 16:25:49 +00:00
Robert Schulze
c9ce0efa66
Instantiate MultiMatchAnyImpl template using enums 
- With this, invalid combinations of the FindAny/FindAnyIndex bools are
  no longer possible and we can remove the corresponding check

- Also makes the instantiations more readable.
 2022-06-26 16:25:49 +00:00
Robert Schulze
2f15d45f27
Move check for regexp array size into implementations 
- This is not needed for non-const regexp array arguments (the
  cardinality of arrays is fixed per column) but it cleans up the code
  and runs the check only in functions which have restrictions on the
  number of patterns.

- For functions using hyperscans, it was checked that the number of
  regexes is < 2^32. Removed the check because I don't think anyone will
  every specify 4 billion patterns.
 2022-06-26 16:25:43 +00:00
Robert Schulze
3478db9fb6
Move check for regexp array size into implementations 
- This is not needed for non-const regexp array arguments (the
  cardinality of arrays is fixed per column) but it cleans up the code
  and runs the check only in functions which have restrictions on the
  number of patterns.

- For functions using hyperscans, it was checked that the number of
  regexes is < 2^32. Removed the check because I don't think anyone will
  every specify 4 billion patterns.
 2022-06-26 15:38:12 +00:00
Robert Schulze
7913edc172
Move check for hyperscan regexp constraints into implementations 
- This is preparation for non-const regexp arguments, where this check
  will run for each row.
 2022-06-26 15:38:05 +00:00
Robert Schulze
89bfdd50bf
Remove unnecessary check 
- getReturnTypeImpl() ensures that the haystack column has type "String"
  and we can simply assert that.
 2022-06-26 15:34:24 +00:00
Robert Schulze
580d89477f
Minimally faster performance 2022-06-26 15:34:22 +00:00
Robert Schulze
4bc59c18e3
Cosmetics: Move some code around + docs + whitespaces + minor stuff 2022-06-26 15:34:15 +00:00
Robert Schulze
1273756911
Cosmetics: fmt-based exceptions 2022-06-26 15:33:18 +00:00