thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-03 13:02:00 +00:00
Code Issues Packages Projects Releases Wiki Activity
98,182 Commits 9,183 Branches 1,782 Tags 2.2 GiB
8c93a9adda
Commit Graph

31520 Commits

Author SHA1 Message Date
Alexey Milovidov
ada7a44ae4 Remove -WithTerminatingZero methods 2022-09-17 05:34:18 +02:00
Alexey Milovidov
0c3835240b Fix error with multidimensional arrays 2022-09-17 04:16:24 +02:00
Alexey Milovidov
b60a5a5bc5 Merge branch 'master' into fix-getdataat-multidimensional-arrays 2022-09-17 04:07:12 +02:00
Alexey Milovidov
35cce03125 Remove dlopen 2022-09-17 03:02:34 +02:00
Alexey Milovidov
31ebd34f5b
Merge pull request #41193 from azat/DWARF-5-parser-v3 
Add support for DWARF-5 (without emitting them in binaries)
 2022-09-17 02:58:02 +03:00
Alexey Milovidov
c09604bef2
Merge pull request #41363 from kitaisreal/small-style-fixes 
Small style fixes
 2022-09-17 00:19:00 +03:00
Denny Crane
6540d55d42
Merge branch 'master' into test/zvonand-monotonic 2022-09-16 17:02:09 -03:00
Nikita Taranov
6f186d3dd2
Do not return empty blocks from ConvertingAggregatedToChunksTransform (#41152) 
* impl

* add test

* update test
 2022-09-16 21:54:36 +02:00
avogar
fa172e21ae Remove unnecessary method implementation 2022-09-16 17:47:05 +00:00
Igor Nikonov
24de5530d6 Remove unnecessary headers 2022-09-16 17:16:32 +00:00
Igor Nikonov
b34ec6453b Fix review comments 
+ one more test
 2022-09-16 17:11:08 +00:00
Vitaly Baranov
391507b4b6 Update src/Access/DiskAccessStorage.cpp 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2022-09-16 19:02:06 +02:00
Vitaly Baranov
2ec6ef497d Make DiskAccessStorage::insertWithID() more consistent with MemoryAccessStorage::insertWithID(). 2022-09-16 19:02:06 +02:00
Vitaly Baranov
7e716d14cd Make ReplicatedAccessStorage::insertWithID() more consistent with MemoryAccessStorage::insertWithID(). 2022-09-16 19:02:06 +02:00
Vitaly Baranov
646cd55690 Improve recovery of ReplicatedAccessStorage after errors. 2022-09-16 19:02:06 +02:00
Vitaly Baranov
8f9f5c69da Use MemoryAccessStorage to simplify ReplicatedAccessStorage. 2022-09-16 19:02:06 +02:00
Dmitry Novik
eacc790050
Merge branch 'master' into fix-odr-vialation 2022-09-16 18:44:53 +02:00
avogar
c9a9ef5b7e Fix comments 2022-09-16 16:12:30 +00:00
Alexander Tokmakov
65474830ac make tryEnqueueReplicatedDDL virtual 2022-09-16 16:25:32 +02:00
Nikolai Kochetov
16f78eb804
Merge pull request #41281 from ClickHouse/fix-key-condition-with-actions-dag-literal-name 
Fix wrong literal name in KeyCondition using ActionsDAG.
 2022-09-16 16:06:38 +02:00
Kruglov Pavel
2d4a6b38af
Merge branch 'master' into fix-totals-extremes 2022-09-16 15:03:59 +02:00
alesapin
d9c3549cc8 Revert s3 prefixes + add retries on write requests unexpected errors 2022-09-16 14:43:43 +02:00
alesapin
456da30c9f Revert "Merge pull request #41249 from ClickHouse/revert-40968-s3-sharding-2" 
This reverts commit ac60300997, reversing
changes made to a9c272283a.
 2022-09-16 14:06:26 +02:00
alesapin
06ccdb1ecf Fix nasty bug 2022-09-16 13:49:39 +02:00
Alexander Tokmakov
9597a47016
Merge pull request #41379 from ClickHouse/tavplubix-patch-5 
Fix an assertion in `wasInvolvedInTransaction`
 2022-09-16 14:24:26 +03:00
Robert Schulze
b32b02d844
Merge pull request #40897 from ClickHouse/catboost-bridge-resurrected 
Move CatBoost evaluation into clickhouse-library-bridge
 2022-09-16 13:12:09 +02:00
Anton Popov
5584ef71f6
Merge pull request #41388 from den-crane/bug/optimize_rewrite_sum_if_to_count_if 
Disable optimize_rewrite_sum_if_to_count_if by default
 2022-09-16 13:05:23 +02:00
Igor Nikonov
eeecaf7a31 Merge remote-tracking branch 'origin/master' into distinct_in_order_wo_order_by 2022-09-16 10:30:52 +00:00
Igor Nikonov
ba1f7de243 Fix review comments 2022-09-16 10:30:30 +00:00
Antonio Andelic
acd9cb089e Fix segfault during shutdown when RAFT server failed to start 2022-09-16 10:03:21 +00:00
Roman Vasin
bc97da686a Add const for bool enable_extended_results_for_datetime_functions 2022-09-16 09:54:24 +00:00
alesapin
af259a2a68
Update src/Storages/MergeTree/MergeFromLogEntryTask.cpp 
Co-authored-by: Alexander Tokmakov <tavplubix@clickhouse.com>
 2022-09-16 11:50:07 +02:00
Roman Vasin
d82cd7b007 Add many cosmetic changes to C++ code and MD docs 2022-09-16 09:37:30 +00:00
Roman Vasin
7e733887b9 Add narrowing for toStartOfISOYear; improve 02403_enable_extended_results_for_datetime_functions 2022-09-16 08:40:15 +00:00
Alexey Milovidov
701fdd77b4 Fix wrong implementation of getDataAt for multidimensional arrays 2022-09-16 08:12:28 +02:00
Antonio Andelic
222b964070
Merge pull request #38200 from ClickHouse/keeper-linearizable-reads 
Keeper faster linearizable reads
 2022-09-16 07:58:52 +02:00
Alexey Milovidov
da01982652
Merge pull request #41046 from azat/build/llvm-15 
Switch to llvm/clang 15
 2022-09-16 07:31:06 +03:00
Raúl Marín
c3ff66bd9d
Implement batch processing for aggregate functions with multiple nullable arguments (#41058) 
* Implement batch processing for aggregate functions with multiple nullable arguments

* Fix broken perf test

* Improve filter handling in addBatchSinglePlace with nullable arguments

* Fix detecting the Null filter usage
 2022-09-15 23:51:38 +02:00
Raúl Marín
6dac509739
Speed up reading uniqState (#41089) 
* Speed up reading UniquesHashSet

* Improve uniq serialization tests
 2022-09-15 23:41:15 +02:00
Igor Nikonov
8f500019eb Merge remote-tracking branch 'origin/master' into distinct_in_order_wo_order_by 2022-09-15 21:05:59 +00:00
Alexander Tokmakov
8301f5758b Update IMergeTreeDataPart.cpp 2022-09-15 21:32:57 +02:00
Alexander Tokmakov
03c193ccca fix ON CLUSTER with Replicated database cluster 2022-09-15 21:15:57 +02:00
Denny Crane
7cfdfbe787 disable optimize_rewrite_sum_if_to_count_if by default 2022-09-15 15:30:24 -03:00
Alexander Tokmakov
32391f737a Merge branch 'master' into fix_41209 2022-09-15 20:27:51 +02:00
avogar
ef7f7b30aa Fix tests 2022-09-15 18:20:17 +00:00
avogar
92979af3bf Add missing methods in combinators 2022-09-15 17:45:50 +00:00
alesapin
82611d891b
Merge branch 'master' into fix-odr-vialation 2022-09-15 19:33:09 +02:00
alesapin
be81d653e1 Fix comment 2022-09-15 18:56:16 +02:00
alesapin
b098c1be36 Fix data duplication in zero copy replication 2022-09-15 18:54:27 +02:00
Antonio Andelic
fe78c063f6
Merge branch 'master' into keeper-linearizable-reads 2022-09-15 18:40:23 +02:00
Antonio Andelic
39307f9dba Extract into function 2022-09-15 16:14:53 +00:00
Antonio Andelic
0dbd0c1f72 Merge branch 'master' into keeper-linearizable-reads 2022-09-15 16:03:18 +00:00
Anton Popov
b5e9884355 fix subcolumns tree 2022-09-15 14:17:10 +00:00
Roman Vasin
5c242de43a Merge branch 'master' of github.com:ClickHouse/ClickHouse into ADQM-528-B 2022-09-15 13:47:51 +00:00
alesapin
7570c5cf67 Remove recursive without blobs 2022-09-15 15:31:19 +02:00
alesapin
c585ed47fc Forget to remove 2022-09-15 15:15:28 +02:00
serxa
b3b8123f73 fix style 2022-09-15 12:53:56 +00:00
alesapin
e127c8b4a9 Add kringe 2022-09-15 14:49:31 +02:00
alesapin
c04f7afb2a Delete kringe 2022-09-15 14:48:59 +02:00
Roman Vasin
563449779c Replace enable_date32_results by enable_extended_results_for_datetime_functions 2022-09-15 12:48:09 +00:00
Alexander Tokmakov
b26cc5a1ba
Merge pull request #41296 from young-scott/fix_issuse_41096 
fix issue #41096
 2022-09-15 15:44:05 +03:00
avogar
6bb6c9e916 Fix comment 2022-09-15 12:41:39 +00:00
avogar
d41bb7c2a4 Fix comment 2022-09-15 12:40:32 +00:00
Alexander Tokmakov
77c4ea9647 fix 2022-09-15 14:34:31 +02:00
Denny Crane
1b8b2ebed5
Disable optimize_monotonous_functions_in_order_by by default (#41136) 
* #40094 disable optimize_monotonous_functions_in_order_by by default

* fix 01576_alias_column_rewrite test

* fix incorrect 02149_read_in_order_fixed_prefix.sql test

* Update 02149_read_in_order_fixed_prefix.sql
 2022-09-15 13:41:46 +02:00
Azat Khuzhin
5cc5ca22de Fix possible deadlock with async_socket_for_remote/use_hedged_requests and parallel KILL 
Right now it is possible to call QueryStatus::addPipelineExecutor() when
the executors_mutex already acquired, it is possible when the query was
cancelled via KILL QUERY.

Here I will show some traces from debugger from a real example, where
tons of ProcessList::insert() got deadlocked.

Let's look at the lock owner for one of the threads that was deadlocked
in ProcessList::insert():

    (gdb) p *mutex
    $2 = {
      __data = {
        __owner = 46899,
      },
    }

And now let's see the stack trace of the 46899:

    #0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:103
    #1  0x00007fb65569b714 in __GI___pthread_mutex_lock (mutex=0x7fb4a9d15298) at ../nptl/pthread_mutex_lock.c:80
    #2  0x000000001b6edd91 in pthread_mutex_lock (arg=0x7fb4a9d15298) at ../src/Common/ThreadFuzzer.cpp:317
    #3  std::__1::__libcpp_mutex_lock (__m=0x7fb4a9d15298) at ../contrib/libcxx/include/__threading_support:303
    #4  std::__1::mutex::lock (this=0x7fb4a9d15298) at ../contrib/libcxx/src/mutex.cpp:33
    #5  0x0000000014c7ae63 in std::__1::lock_guard<std::__1::mutex>::lock_guard (__m=..., this=<optimized out>) at ../contrib/libcxx/include/__mutex_base:91
    #6  DB::QueryStatus::addPipelineExecutor (this=0x7fb4a9d14f90, e=0x80) at ../src/Interpreters/ProcessList.cpp:372
    #7  0x0000000015bee4a7 in DB::PipelineExecutor::PipelineExecutor (this=0x7fb4b1e53618, processors=..., elem=<optimized out>) at ../src/Processors/Executors/PipelineExecutor.cpp:54
    #12 std::__1::make_shared<DB::PipelineExecutor, std::__1::vector<std::__1::shared_ptr<DB::IProcessor>, std::__1::allocator<std::__1::shared_ptr<DB::IProcessor> > >&, DB::QueryStatus*&, void> (__args=@0x7fb63095b9b0: 0x7fb4a9d14f90, __args=@0x7fb63095b9b0: 0x7fb4a9d14f90) at ../contrib/libcxx/include/__memory/shared_ptr.h:963
    #13 DB::QueryPipelineBuilder::execute (this=0x7fb63095b8b0) at ../src/QueryPipeline/QueryPipelineBuilder.cpp:552
    #14 0x00000000158c6c27 in DB::Connection::sendExternalTablesData (this=0x7fb6545e9d98, data=...) at ../src/Client/Connection.cpp:797
    #27 0x0000000014043a81 in DB::RemoteQueryExecutorRoutine::operator() (this=0x7fb63095bf20, sink=...) at ../src/QueryPipeline/RemoteQueryExecutorReadContext.cpp:46
    #32 0x000000000a16dd4f in make_fcontext () at ../contrib/boost/libs/context/src/asm/make_x86_64_sysv_elf_gas.S:71

And also in the logs you can see very strange things for this thread:

    2022.09.13 14:14:51.228979 [ 51145 ] {1712D4E914EC7C99} <Debug> Connection (localhost:9000): Sent data for 1 external tables, total 11 rows in 0.00046389 sec., 23688 rows/sec., 3.84 KiB (8.07 MiB/sec.), compressed 1.1070121092649958 times to 3.47 KiB (7.29 MiB/sec.)
    ...
    2022.09.13 14:14:51.719402 [ 46899 ] {7c90ffa4-1dc8-42fd-938c-4e307c244394} <Debug> executeQuery: (from 10.101.15.181:42478) KILL QUERY WHERE query_id = '1712D4E914EC7C99' (stage: Complete)
    2022.09.13 14:14:51.719488 [ 46899 ] {7c90ffa4-1dc8-42fd-938c-4e307c244394} <Debug> executeQuery: (internal) SELECT query_id, user, query FROM system.processes WHERE query_id = '1712D4E914EC7C99' (stage: Complete)
    2022.09.13 14:14:51.719754 [ 46899 ] {7c90ffa4-1dc8-42fd-938c-4e307c244394} <Trace> ContextAccess (default): Access granted: SELECT(user, query_id, query) ON system.processes
    2022.09.13 14:14:51.720544 [ 46899 ] {7c90ffa4-1dc8-42fd-938c-4e307c244394} <Trace> InterpreterSelectQuery: FetchColumns -> Complete
    2022.09.13 14:14:53.228964 [ 46899 ] {7c90ffa4-1dc8-42fd-938c-4e307c244394} <Debug> Connection (localhost:9000): Sent data for 2 scalars, total 2 rows in 2.6838e-05 sec., 73461 rows/sec., 68.00 B (2.38 MiB/sec.), compressed 0.4594594594594595 times to 148.00 B (5.16 MiB/sec.)

How is this possible? The answer is fibers and query cancellation
routine. During cancellation of async queries it going into fibers again
and try to do this gracefully. However because of this during canceling
query it may call QueryStatus::addPipelineExecutor() from
QueryStatus::cancelQuery().

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-09-15 14:41:27 +03:00
serxa
4971c1f04d review fixes 2022-09-15 11:34:14 +00:00
Maksim Kita
276d3628e9 Small style fixes 2022-09-15 13:05:18 +02:00
Roman Vasin
3346d8e27b Add many cosmetic changes to C++ code 2022-09-15 10:56:05 +00:00
Igor Nikonov
da75d28019 Merge remote-tracking branch 'origin/master' into distinct_in_order_wo_order_by 2022-09-15 10:54:42 +00:00
Kseniia Sumarokova
9e37a6356f
Merge pull request #41297 from kssenii/fix-logical-error 
Fix incorrect logical error in disk object storage
 2022-09-15 12:00:53 +02:00
Kseniia Sumarokova
69f38a6228
Merge pull request #41330 from kssenii/fix-composable-cache-setting-incompatibility 
Fix incompatibility of cache after switching setting `do_no_evict_...` from 1 to 0, 0 to 1
 2022-09-15 12:00:22 +02:00
Kruglov Pavel
73cf72a5a4
Merge pull request #41309 from Avogar/fix-msgpack 
Add column type check before UUID insertion in MsgPack format
 2022-09-15 11:37:57 +02:00
Kruglov Pavel
982b9baeec
Fix 2022-09-15 11:32:14 +02:00
Robert Schulze
b64af5989e
Merge branch 'master' into opentelemetry_stress 2022-09-15 09:43:11 +02:00
Anton Popov
66667737f0 fix writing of empty columns of type Object 2022-09-15 03:38:22 +00:00
Kseniia Sumarokova
780c0e9b2f
Update CachedOnDiskReadBufferFromFile.cpp 2022-09-15 00:18:23 +02:00
Kseniia Sumarokova
ebf7cb6338
Merge pull request #41308 from ClickHouse/kssenii-patch-5 
Fix after #41042
 2022-09-15 00:15:54 +02:00
Igor Nikonov
f011f72114 Try to apply optimization only if optimizer reaches pre distinct node 
+ more comments
 2022-09-14 20:55:49 +00:00
Kseniia Sumarokova
0b1c2c62fd
Update CachedOnDiskReadBufferFromFile.cpp 2022-09-14 20:59:45 +02:00
kssenii
559c696230 Fix 2022-09-14 20:35:28 +02:00
avogar
f1a0501eb2 Fix memory leaks and segfaults in combinators 2022-09-14 18:01:49 +00:00
Igor Nikonov
bab5744a1d Merge remote-tracking branch 'origin/master' into distinct_in_order_wo_order_by 2022-09-14 17:54:46 +00:00
Igor Nikonov
d91ea0c5ab Fix + tests 2022-09-14 17:54:05 +00:00
Alexander Tokmakov
173df9a73a
Update StorageSystemDDLWorkerQueue.cpp 2022-09-14 20:38:50 +03:00
Alexander Tokmakov
fc73134145 fix race between DROP and short form of ATTACH 2022-09-14 18:03:36 +02:00
alesapin
63213465ef Fix comment 2022-09-14 17:28:12 +02:00
alesapin
ad279940d6 Trying print less scary messages in case No such key for outdated parts 2022-09-14 17:25:58 +02:00
Robert Schulze
fd97058e45
fix: incorporate review comments 2022-09-14 15:21:24 +00:00
Anton Popov
3623807527
Merge pull request #41290 from CurtizJ/dynamic-columns-20 
Fix `ALTER ADD COLUMN`  with columns of type `Object`
 2022-09-14 14:15:29 +02:00
Anton Popov
defd393fdc
Update src/DataTypes/ObjectUtils.h 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2022-09-14 14:07:07 +02:00
mateng0915
31a0900756 deleted local file 2022-09-14 19:19:47 +08:00
mateng0915
f7f976e94e remove extra space 2022-09-14 19:19:47 +08:00
mateng0915
f38e262706 fixed the unnecessory changes 2022-09-14 19:19:47 +08:00
mateng0915
5badb1b186 resolve the review comments 2022-09-14 19:19:40 +08:00
teng.ma
a85c917488 fix master conflict 2022-09-14 19:18:31 +08:00
teng.ma
ebea9f9359 Feature: Support Event Tracing when calling Aws S3 API 
Description:
============
In  Computing && Storage Architecture,  using S3 as remote / Shared storage, the Method to access S3 is using AWS S3 API
There is a gap between ClickHouse DB with Ozone Operation
In ClickHouse, operation is SQL and background task
In S3 , the operation should be AWS S3 API
And one sql maybe can mapped to multiple  API

Solution:
=========
Added Calling API as event into system.events table
 2022-09-14 19:18:31 +08:00
avogar
59e7eb084c Add column type check before UUID insertion in MsgPack format 2022-09-14 11:15:10 +00:00
Kseniia Sumarokova
cc639f2e2d
Update CachedOnDiskReadBufferFromFile.cpp 2022-09-14 13:13:02 +02:00
Kruglov Pavel
3396ff6c3a
Merge pull request #40516 from zjial/record_errors_for_import_by_csv 
Record errors while reading text formats (CSV, TSV).
 2022-09-14 12:52:32 +02:00
alesapin
51a302a70f
Merge branch 'master' into revert-40968-s3-sharding-2 2022-09-14 11:53:14 +02:00
Kseniia Sumarokova
52db0e5c40
Update DiskObjectStorageMetadata.cpp 2022-09-14 11:52:23 +02:00
young scott
b6888022cd
Merge branch 'master' into fix_issuse_41096 2022-09-14 17:38:22 +08:00
kssenii
ea31302071 Fix 2022-09-14 11:30:06 +02:00
young scott
bc111b5680 fix issuse#41096 2022-09-14 09:15:40 +00:00
Vladimir C
00d0b37cd0
Merge pull request #41263 from ClickHouse/vdimir/quorum-log-msg-fix 2022-09-14 10:54:21 +02:00
Anton Popov
ee54451741 fix adding of columns of type Object 2022-09-13 22:43:59 +00:00
Alexander Tokmakov
cf7d928554
Update PredicateRewriteVisitor.cpp 2022-09-13 22:38:55 +03:00
Robert Schulze
e8e6dddc76
Merge pull request #41034 from FrankChen021/distributed 
Improve the observability of INSERT on distributed table
 2022-09-13 20:55:11 +02:00
Nikolai Kochetov
3f0bc94d25 Comment. 2022-09-13 18:50:01 +00:00
Nikolai Kochetov
66d9d92de4 Fix wrong literal name in KeyCondition using ActionsDAG. 2022-09-13 18:47:29 +00:00
Alexander Tokmakov
0175f3d0d7 fix WHERE pushdown when subquery has OFFSET 2022-09-13 20:44:45 +02:00
Igor Nikonov
c1dc77cd50 Merge remote-tracking branch 'origin/master' into distinct_in_order_wo_order_by 2022-09-13 18:24:48 +00:00
Igor Nikonov
1be5a79e9f Exclude const columns in distinct when matching with sorting key 
+ tests
 2022-09-13 18:23:55 +00:00
Igor Nikonov
c04d46bc96 Update data stream properties after reading in order applied 
+ tests
 2022-09-13 17:50:43 +00:00
Vladimir C
f0e4088c84
Merge pull request #41154 from azat/fix-sorted-block-writer 2022-09-13 19:14:45 +02:00
Antonio Andelic
36f46891dd
Merge pull request #41215 from ClickHouse/keeper-close-connection-on-shutdown 
Close sessions on Keeper shutdown
 2022-09-13 19:09:34 +02:00
Dmitry Novik
7e3fb0a681
Merge pull request #40762 from ClickHouse/grouping-comp 
Fix GROUPING function SQL compatibility
 2022-09-13 17:03:06 +02:00
Dmitry Novik
990ce888bf
Merge branch 'master' into fix-odr-vialation 2022-09-13 16:53:57 +02:00
vdimir
f2acb53887
Better log message for replicas number in StorageReplicatedMerge 2022-09-13 13:49:51 +00:00
Kruglov Pavel
110be0688e
Merge pull request #40909 from ClickHouse/Avogar-patch-1 
Make better exception message in schema inference
 2022-09-13 14:44:29 +02:00
Kruglov Pavel
faa62f96ae
Merge branch 'master' into new-json-formats 2022-09-13 14:41:13 +02:00
Kruglov Pavel
17621b5607
Merge branch 'master' into fix-totals-extremes 2022-09-13 14:36:31 +02:00
Igor Nikonov
f99f57aed8 Remove unrelated changes 2022-09-13 12:24:46 +00:00
Antonio Andelic
baf7255cff
Merge pull request #39976 from ClickHouse/keeper-storage 
KeeperMap storage engine
 2022-09-13 14:15:09 +02:00
Anton Popov
1989380485
Merge pull request #41122 from lthaooo/optimize_compact_part_mutation 
optimize compact part mutation
 2022-09-13 14:10:21 +02:00
Igor Nikonov
d8acb0e7ad Tiny improvements along the code 
- less allocations (potentially)
 2022-09-13 12:06:56 +00:00
zhongyuankai
f36d70e9dd
Merge branch 'master' into compress_marks_and_primary_key 2022-09-13 20:05:39 +08:00
Alexander Tokmakov
4d146b05a9
Merge pull request #38262 from PolyProgrammist/fix-ordinary-system-unfreeze 
Fix SYSTEM UNFREEZE for ordinary database
 2022-09-13 14:55:11 +03:00
Alexander Tokmakov
2845494624
Merge pull request #41120 from azat/async-insert-queue-term 
Terminate AsynchronousInsertQueue w/o waiting for timeout
 2022-09-13 14:52:50 +03:00
Alexander Tokmakov
54001d3871
Revert "Fix uncaught DNS_ERROR on failed connection to replicas" 2022-09-13 13:51:09 +03:00
Vladimir C
cc7292df74
Merge pull request #41105 from ClickHouse/vdimir/upd_storage_rdb_sample_block 2022-09-13 12:37:12 +02:00
Kseniia Sumarokova
b2c9c04c7b
Merge pull request #40821 from kssenii/improve-marks-cache-loading 
Allow to load marks in threadpool in advance
 2022-09-13 12:31:58 +02:00
alesapin
9af591a328
Revert "Sharding s3 key names (2)" 2022-09-13 12:29:02 +02:00
Frank Chen
d3265150c0 Make sure span is finished in the onFinish callback 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-09-13 18:28:21 +08:00
Frank Chen
7303ae1796 Make sure span holder will be destructed only once 2022-09-13 18:27:29 +08:00
Kseniia Sumarokova
3dd28bb79a
Merge pull request #41042 from kssenii/refactor-cache-for-async-download 
Cache refactoring for async download
 2022-09-13 12:26:35 +02:00
Kseniia Sumarokova
5a76440078
Merge pull request #41232 from kssenii/fix-logical-error-in-write-through-cache 
Fix logical error in write-through cache found by stress test
 2022-09-13 12:25:21 +02:00
Kseniia Sumarokova
651d850652
Merge pull request #41230 from kssenii/minor-change 
minor change in MergeTreeDataPartCompact
 2022-09-13 12:23:18 +02:00
Kseniia Sumarokova
6655f1933a
Merge pull request #41233 from josemimbrero-tinybird/fix-filesystem_cache_log 
Fix String log entry in system.filesystem_cache_log
 2022-09-13 12:21:18 +02:00
alesapin
69d08b1007
Merge pull request #41075 from ClickHouse/keeper-fix-possible-segfault 
Fix possible segfault during Keeper shutdown
 2022-09-13 12:13:24 +02:00
Kruglov Pavel
7d7606a1e3
Merge pull request #41162 from Avogar/fix-http-redirect 
Support relative path in Location header after http redirect
 2022-09-13 12:10:44 +02:00
Antonio Andelic
a6b5ffec5d Polishing 2022-09-13 09:51:46 +00:00
Igor Nikonov
c129b31d7b Test fix 2022-09-13 09:40:23 +00:00
Igor Nikonov
4cd35601d5 Merge remote-tracking branch 'origin/master' into distinct_in_order_wo_order_by 2022-09-13 08:19:59 +00:00
zhenjial
5841d9e9b0 sync before destruct 2022-09-13 15:53:24 +08:00
zhenjial
67c08e3e22 sync before destruct 2022-09-13 15:06:22 +08:00
zhenjial
16c8cd0bd3 wait write finish 2022-09-13 14:19:40 +08:00
zhongyuankai
75c0854a78 fix test 2022-09-13 11:18:25 +08:00
zhongyuankai
1477bb132f
Merge branch 'master' into compress_marks_and_primary_key 2022-09-13 11:05:57 +08:00
Igor Nikonov
1c9c303cce Fix typos 2022-09-12 21:11:05 +00:00
Igor Nikonov
7502be3b75 Fixes 2022-09-12 20:11:24 +00:00
Sergei Trifonov
cad3a6b764 fix style 2022-09-12 22:01:45 +02:00
Robert Schulze
fac1be9700
chore: restore SYSTEM RELOAD MODEL(S) and moniting view SYSTEM.MODELS 
- This commit restores statements "SYSTEM RELOAD MODEL(S)" which provide
  a mechanism to update a model explicitly. It also saves potentially
  unnecessary reloads of a model from disk after it's initial load.

  To keep the complexity low, the semantics of "SYSTEM RELOAD MODEL(S)
  was changed from eager to lazy. This means that both statements
  previously immedately reloaded the specified/all models, whereas now
  the statements only trigger an unload and the first call to
  catboostEvaluate() does the actual load.

- Monitoring view SYSTEM.MODELS is also restored but with some obsolete
  fields removed. The view was not documented in the past and for now it
  remains undocumented. The commit is thus not considered a breach of
  ClickHouse's public interface.
 2022-09-12 19:33:02 +00:00
Sergei Trifonov
c31818260f renames and refactoring 2022-09-12 21:03:06 +02:00
Kseniia Sumarokova
5d1e9822f8
Merge branch 'master' into fix-filesystem_cache_log 2022-09-12 20:32:42 +02:00
Antonio Andelic
583c530d0f
Merge pull request #41177 from azat/http-handlers-reload 
Apply changes to http handlers on fly without server restart
 2022-09-12 20:25:00 +02:00
Alexander Tokmakov
05b4303234
Merge pull request #41146 from ClickHouse/followup_39933 
Stop background operations before converting database from Ordinary
 2022-09-12 21:20:25 +03:00
Antonio Andelic
b2cc6a8cc6 Use promise/future 2022-09-12 18:19:41 +00:00
Kruglov Pavel
702ddff5f6
Fix style 2022-09-12 19:38:34 +02:00
Kruglov Pavel
060adfbe93
Merge branch 'master' into new-json-formats 2022-09-12 19:37:46 +02:00
avogar
8ac2fc7b26 Don't outout totals/extremes in all row formats, update docs 2022-09-12 17:21:40 +00:00
Igor Nikonov
bdd7c7c2c1 Merge remote-tracking branch 'origin/master' into distinct_in_order_wo_order_by 2022-09-12 17:02:23 +00:00
Igor Nikonov
9fc0aaf477 Something working with tests 2022-09-12 17:01:26 +00:00
Jose
1880a745b2 Fix String in log entry 2022-09-12 18:53:54 +02:00
kssenii
a07f3d4bea Fix 2022-09-12 18:45:48 +02:00
Frank Chen
20191932df Fix style 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-09-13 00:41:05 +08:00
kssenii
06f80d78f8 Fix 2022-09-12 18:16:53 +02:00
Frank Chen
7e1f2901da Fix 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-09-13 00:06:17 +08:00
Frank Chen
7d6903bdc0 Fix 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-09-13 00:06:03 +08:00
Azat Khuzhin
b698a4ff65 Apply changes to http handlers on fly without server restart 
This has been implemented by simply restarting http servers in case of
http_handlers directive in configuration xml had been changed.

But, for this I have to change the handlers interface to accept
configuration separatelly, since the configuration that contains in the
server is the configuration with which server had been started.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Co-authored-by: Antonio Andelic
 2022-09-12 17:34:51 +02:00
vdimir
ba506043ce
Fix exponential query rewrite in case of lots of cross joins with where 2022-09-12 15:05:34 +00:00
Alexander Tokmakov
da4fbb552d
Merge branch 'master' into fix_parts_removal 2022-09-12 17:49:29 +03:00
Antonio Andelic
deee0d639f Address PR comments 2022-09-12 14:43:52 +00:00
Alexey Milovidov
2aedd41023
Remove strange code (#40195) 
* Remove strange code

* Even more code removal

* Fix style

* Remove even more code

* Simplify code by making it slower

* Attempt to do something

* Attempt to do something

* Well do something with this horrible trash

* Add a test
 2022-09-12 16:29:23 +02:00
Sergei Trifonov
fe88a7991d
Merge branch 'master' into readonly-settings-allow 2022-09-12 16:20:40 +02:00
Frank Chen
ebaa24ecae Fix flaky tests 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-09-12 22:15:30 +08:00
Azat Khuzhin
5cb37eb1a0 Terminate AsynchronousInsertQueue w/o waiting for timeout 
Previously it simply sleeps for async_insert_busy_timeout_ms delay on
shutdown, which is not great, and produces false-positive hung checks on
CI [1].

  [1]: https://pastila.nl/?0087a100/db80915eac58e67eb05516bf459e5510

Refs: #29390 (cc @tavplubix, @CurtizJ)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-09-12 16:11:54 +02:00
kssenii
413fbb6507 Self-review fixes 2022-09-12 15:46:11 +02:00
Antonio Andelic
1cd43d8e5c Merge branch 'master' into keeper-storage 2022-09-12 13:42:51 +00:00
Anton Popov
76af7efba9
Merge branch 'master' into optimize_compact_part_mutation 2022-09-12 15:35:47 +02:00
Dmitry Novik
c236706703
Merge branch 'master' into grouping-comp 2022-09-12 15:19:51 +02:00
Antonio Andelic
c2dfabe51c Use shared_ptr for Event 2022-09-12 12:25:39 +00:00
Antonio Andelic
e985ee3354 Close sessions on Keeper shutdown 2022-09-12 12:22:48 +00:00
Alexander Tokmakov
ed3c95c28f review fix 2022-09-12 13:53:43 +02:00
Sergei Trifonov
4810a59866 fix uninitialized variables 2022-09-12 13:27:06 +02:00
Kseniia Sumarokova
df22e8d30c
Merge pull request #41190 from kssenii/fix-clean-up-thread 
Fix cleaning up broken detached part after timeout
 2022-09-12 13:21:45 +02:00
kssenii
88b589917c Update defaults 2022-09-12 12:48:47 +02:00
Kruglov Pavel
6535301888
Merge branch 'master' into Avogar-patch-1 2022-09-12 12:23:28 +02:00
Alexander Tokmakov
7f462a8762
Update src/Storages/StorageReplicatedMergeTree.cpp 2022-09-12 13:22:08 +03:00
Kseniia Sumarokova
d0c058f273
Merge pull request #41132 from Algunenano/views_log_window 
Fix query_views_log with Window views
 2022-09-12 11:56:22 +02:00
Alexey Milovidov
9c5f9f1815
Merge pull request #41158 from den-crane/bug/trivial_count_optimization_with_array_join 
Fix: trivial count optimization with array join. revert revert #39466
 2022-09-12 08:56:10 +03:00
Alexey Milovidov
94d6e9ce7d
Merge pull request #41189 from kssenii/fix-positional-args 
Fix for positional arguments
 2022-09-12 03:21:39 +03:00
Azat Khuzhin
7e130aeb69 Add support for DWARF-5 (without emitting them in binaries) 
ClickHouse changes to the folly parser:
- use camel_case
- add NOLINT
- avoid using folly:: (use std:: instead)
- avoid using boost:: (use std:: instead)

But note, now it has not been enabled by default (like it was
initially), because you may need recent debugger to support DWARF-5
correctly, and to make debugging easier, let's do this later.

A good example is gdb 10, even though it looks like it should support
it, it still produce some errors, like here [1]:

    Dwarf Error: DW_FORM_strx1 found in non-DWO CU [in module /usr/bin/clickhouse]

  [1]: https://github.com/ClickHouse/ClickHouse/pull/40772#issuecomment-1236331323

And not only it complains, apparently it can "activate" SDT probes
(replace "nop" with "int3"), and I believe this is what happens here
[2].

  [2]: https://github.com/ClickHouse/ClickHouse/pull/41063#issuecomment-1242992314

There you got int3 in the case when ClickHouse got SIGTRAP:

<details>

```
    0x7f494705e093 <+1139>: jne    0x7f494705e450            ; <+2096> [inlined] update_tls_slotinfo at dl-open.c:732
    0x7f494705e099 <+1145>: testl  %r13d, %r13d
    0x7f494705e09c <+1148>: je     0x7f494705e09f            ; <+1151> at dl-open.c:744:6
    0x7f494705e09e <+1150>: int3
->  0x7f494705e09f <+1151>: movl   -0x54(%rbp), %eax
    0x7f494705e0a2 <+1154>: testl  %eax, %eax
    0x7f494705e0a4 <+1156>: jne    0x7f494705e410            ; <+2032> at dl-open.c:745:5

But if I repeat the query it does not:

    0x7ffff7fe5093 <+1139>: jne    0x7ffff7fe5450            ; <+2096> [inlined] update_tls_slotinfo at dl-open.c:732
    0x7ffff7fe5099 <+1145>: testl  %r13d, %r13d
    0x7ffff7fe509c <+1148>: je     0x7ffff7fe509f            ; <+1151> at dl-open.c:744:6
    0x7ffff7fe509e <+1150>: nop
->  0x7ffff7fe509f <+1151>: movl   -0x54(%rbp), %eax
    0x7ffff7fe50a2 <+1154>: testl  %eax, %eax
    0x7ffff7fe50a4 <+1156>: jne    0x7ffff7fe5410            ; <+2032> at dl-open.c:745:5
```

</details>

Test command was:

    clickhouse local --stacktrace -q "select * from file('data.capnp', 'CapnProto', 'val1 char') settings format_schema='nonexist:Message'

*P.S. I did this, because I have libraries compiled with DWARF5 (i.e. glibc), and dwarf parser simply fails on my dev env.*

Refs: 490b287ca3
(cherry picked from commit ee5696bb32)
(cherry picked from commit e03870bc8b)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-09-11 21:06:14 +02:00
Alexey Milovidov
8b5328f7e5 Merge branch 'master' into fix-trash-base58 2022-09-11 20:43:54 +02:00
Alexey Milovidov
3e6c4e3f16 Fix clang-tidy 2022-09-11 20:43:36 +02:00
Alexey Milovidov
9a0892c40c
Merge pull request #41171 from ClickHouse/fix-half-of-trash 
Fix half of trash
 2022-09-11 21:40:22 +03:00
Alexey Milovidov
bd40d0de7d
Merge pull request #41187 from ClickHouse/faster-build 
Faster build
 2022-09-11 21:39:36 +03:00
Denny Crane
0e4e868f29
Merge branch 'master' into bug/trivial_count_optimization_with_array_join 2022-09-11 15:23:10 -03:00
kssenii
9c2bbcf4a6 Fix 2022-09-11 17:38:51 +02:00
kssenii
02c3d8b0a0 Fix 2022-09-11 16:42:56 +02:00
alesapin
6eead4ea1e Fix fasttest build 2022-09-11 15:16:54 +02:00
kssenii
2f3bfc5411 Fix 2022-09-11 13:54:25 +02:00
alesapin
21ab379026 Fix comment 2022-09-11 13:47:04 +02:00
alesapin
ac53830000 Fix hung check 2022-09-11 13:37:39 +02:00
kssenii
b7d751b782 Merge remote-tracking branch 'upstream/master' into improve-marks-cache-loading 2022-09-11 13:23:30 +02:00
kssenii
37bdba98b1 Fix 2022-09-11 12:58:04 +02:00
Alexey Milovidov
7f5a1cbe16 Faster build 2022-09-11 09:29:24 +02:00
Alexey Milovidov
5802c2fdd2
Merge pull request #40713 from ClickHouse/remove-useless-line 
Remove one line from XML, because we do not care
 2022-09-11 09:53:30 +03:00
Alexey Milovidov
5592c8e775 Faster build 2022-09-11 08:50:44 +02:00
Alexey Milovidov
88e9178830 Faster build 2022-09-11 08:46:05 +02:00
Alexey Milovidov
0896e6b9b6
Merge pull request #41183 from ClickHouse/fix-parallel-hash-join-for-floats 
Better parallel hash JOIN for floats
 2022-09-11 09:13:57 +03:00
Alexey Milovidov
e0a9ae0496 Fix base58 trash 2022-09-11 08:09:14 +02:00
Alexey Milovidov
3d8a2130cf Continue fixing the trash 2022-09-11 06:31:15 +02:00
Alexey Milovidov
9fbd361a2b Merge branch 'master' into fix-trash-base58 2022-09-11 06:23:01 +02:00
Alexey Milovidov
7f1e7b5967 Merge branch 'master' into fix-half-of-trash 2022-09-11 06:20:47 +02:00
Alexey Milovidov
05d0f48d17
Merge pull request #41173 from ClickHouse/remove-some-methods 
Remove some methods
 2022-09-11 07:19:52 +03:00
Alexey Milovidov
90fd507292 Fix test 2022-09-11 06:18:43 +02:00
Alexey Milovidov
512fb998f1 Fix ridiculous trash in Base58 2022-09-11 03:54:40 +02:00
Alexey Milovidov
cc38ac3764 Better parallel hash JOIN for floats 2022-09-11 03:04:44 +02:00
Alexey Milovidov
ae6c74a916 Merge branch 'master' into fix-half-of-trash 2022-09-11 02:30:56 +02:00
Alexey Milovidov
6f513dab6f
Merge pull request #41169 from ClickHouse/remove-trash-4 
Fix strange code in DatabaseReplicated
 2022-09-11 03:30:08 +03:00
Alexey Milovidov
2b6724c5e3 Fix example 2022-09-11 01:58:55 +02:00
Alexey Milovidov
b75d56ad9b Merge branch 'master' into remove-some-methods 2022-09-11 01:55:02 +02:00
Alexey Milovidov
91338ea771
Merge branch 'master' into remove-useless-line 2022-09-11 02:03:39 +03:00
Azat Khuzhin
c1e70169d2 Suppress clang-analyzer-cplusplus.NewDelete in MsgPackRowInputFormat 
Appartently there is some issue with clang-15, since even the following
example shows error [1].

  [1]: https://gist.github.com/azat/027f0e949ea836fc2e6269113ceb8752

clang-tidy report [1]:

    FAILED: src/CMakeFiles/dbms.dir/Processors/Formats/Impl/MsgPackRowInputFormat.cpp.o                                                                                                            /usr/bin/cmake -E __run_co_compile --launcher="prlimit;--as=10000000000;--data=5000000000;--cpu=1000;/usr/bin/ccache" --tidy=/usr/bin/clang-tidy-15 --source=/ch/src/Processors/Formats/Impl/MsgPackRowInputFormat.cpp -- /usr/bin/clang++-15 --target=x86_64-linux-gnu --sysroot=/ch/cmake/linux/../../contrib/sysroot/linux-x86_64/x86_64-linux-gnu/libc  -DAWS_SDK_VERSION_MAJOR=1 -DAWS_SDK_VERSION_MINOR=7 -DAWS_SDK_VERSION_PATCH=231 -DBOOST_ASIO_HAS_STD_INVOKE_RESULT=1 -DBOOST_ASIO_STANDALONE=1 -DCARES_STATICLIB -DCONFIGDIR=\"\" -DENABLE_MULTITARGET_CODE=1 -DENABLE_OPENSSL_ENCRYPTION -DHAS_RESERVED_IDENTIFIER -DHAVE_CONFIG_H -DLIBSASL_EXPORTS=1 -DLZ4_DISABLE_DEPRECATE_WARNINGS=1 -DOBSOLETE_CRAM_ATTR=1 -DOBSOLETE_DIGEST_ATTR=1 -DPLUGINDIR=\"\" -DPOCO_ENABLE_CPP11 -DPOCO_HAVE_FD_EPOLL -DPOCO_OS_FAMILY_UNIX -DSASLAUTHD_CONF_FILE_DEFAULT=\"\" -DSNAPPY_CODEC_AVAILABLE -DSTD_EXCEPTION_HAS_STACK_TRACE=1 -DUNALIGNED_OK -DWITH_COVERAGE=0 -DWITH_GZFILEOP -DX86_64 -DZLIB_COMPAT -D_LIBCPP_ENABLE_THREAD_SAFETY_ANNOTATIONS -Iincludes/configs -I/ch/src -Isrc -Isrc/Core/include -I/ch/base/glibc-compatibility/memcpy -I/ch/base/base/.. -Ibase/base/.. -I/ch/contrib/cctz/include -I/ch/base/pcg-random/. -I/ch/contrib/miniselect/include -I/ch/contrib/zstd/lib -Icontrib/cyrus-sasl-cmake -I/ch/contrib/lz4/lib -I/ch/src/Common/mysqlxx/. -Icontrib/c-ares -I/ch/contrib/c-ares -I/ch/contrib/c-ares/include -isystem /ch/contrib/libcxx/include -isystem /ch/contrib/libcxxabi/include -isystem /ch/contrib/libunwind/include -isystem /ch/contrib/libdivide/. -isystem /ch/contrib/jemalloc-cmake/include -isystem /ch/contrib/llvm/llvm/include -isystem contrib/llvm/llvm/include -isystem /ch/contrib/abseil-cpp -isystem /ch/contrib/croaring/cpp -isystem /ch/contrib/croaring/include -isystem /ch/contrib/cityhash102/include -isystem /ch/contrib/boost -isystem /ch/contrib/poco/Net/include -isystem /ch/contrib/poco/Foundation/include -isystem /ch/contrib/poco/NetSSL_OpenSSL/include -isystem /ch/contrib/poco/Crypto/include -isystem /ch/contrib/boringssl/include -isystem /ch/contrib/poco/Util/include -isystem /ch/contrib/poco/JSON/include -isystem /ch/contrib/poco/XML/include -isystem /ch/contrib/replxx/include -isystem /ch/contrib/fmtlib-cmake/../fmtlib/include -isystem /ch/contrib/magic_enum/include -isystem /ch/contrib/double-conversion -isystem /ch/contrib/dragonbox/include -isystem /ch/contrib/re2 -isystem contrib/re2-cmake -isystem /ch/contrib/zlib-ng -isystem contrib/zlib-ng-cmake -isystem /ch/contrib/pdqsort -isystem /ch/contrib/xz/src/liblzma/api -isystem /ch/contrib/aws-c-common/include -isystem /ch/contrib/aws-c-event-stream/include -isystem /ch/contrib/aws/aws-cpp-sdk-s3/include -isystem /ch/contrib/aws/aws-cpp-sdk-core/include -isystem contrib/aws-s3-cmake/include -isystem /ch/contrib/snappy -isystem contrib/snappy-cmake -isystem /ch/contrib/msgpack-c/include -isystem /ch/contrib/fast_float/include -isystem /ch/contrib/librdkafka-cmake/include -isystem /ch/contrib/librdkafka/src -isystem contrib/librdkafka-cmake/auxdir -isystem /ch/contrib/cppkafka/include -isystem /ch/contrib/nats-io/src -isystem /ch/contrib/nats-io/src/adapters -isystem /ch/contrib/nats-io/src/include -isystem /ch/contrib/nats-io/src/unix -isystem /ch/contrib/libuv/include -isystem /ch/contrib/krb5/src/include -isystem contrib/krb5-cmake/include -isystem /ch/contrib/NuRaft/include -isystem /ch/contrib/poco/MongoDB/include -isystem contrib/mariadb-connector-c-cmake/include-public -isystem /ch/contrib/mariadb-connector-c/include -isystem /ch/contrib/mariadb-connector-c/libmariadb -isystem /ch/contrib/icu/icu4c/source/i18n -isystem /ch/contrib/icu/icu4c/source/common -isystem /ch/contrib/capnproto/c++/src -isystem /ch/contrib/arrow/cpp/src -isystem /ch/contrib/arrow-cmake/cpp/src -isystem contrib/arrow-cmake/cpp/src -isystem contrib/arrow-cmake/../orc/c++/include -isystem /ch/contrib/orc/c++/include -isystem contrib/avro-cmake/include -isystem /ch/contrib/avro/lang/c++/api -isystem /ch/contrib/openldap-cmake/linux_x86_64/include -isystem /ch/contrib/openldap/include -isystem /ch/contrib/sparsehash-c11 -isystem /ch/contrib/protobuf/src -isystem src/Server/grpc_protos -isystem /ch/contrib/grpc/include -isystem /ch/contrib/libhdfs3/include -isystem /ch/contrib/hive-metastore -isystem /ch/contrib/thrift/lib/cpp/src -isystem contrib/thrift-cmake -isystem /ch/contrib/azure/sdk/core/azure-core/inc-isystem /ch/contrib/azure/sdk/identity/azure-identity/inc -isystem /ch/contrib/azure/sdk/storage/azure-storage-common/inc -isystem /ch/contrib/azure/sdk/storage/azure-storage-blobs/inc -isystem /ch/contrib/s2geometry/src -isystem /ch/contrib/AMQP-CPP/include -isystem /ch/contrib/AMQP-CPP -isystem /ch/contrib/sqlite-amalgamation -isystem /ch/contrib/rocksdb/include -isystem /ch/contrib/libpqxx/include -isystem /ch/contrib/libpq -isystem /ch/contrib/libpq/include -isystem /ch/contrib/libstemmer_c/include -isystem /ch/contrib/wordnet-blast -isystem /ch/contrib/lemmagen-c/include -isystem /ch/contrib/simdjson/include -isystem /ch/contrib/rapidjson/include -isystem /ch/contrib/consistent-hashing --gcc-toolchain=/ch/cmake/linux/../../contrib/sysroot/linux-x86_64 -std=c++20 -fdiagnostics-color=always -Xclang -fuse-ctor-homing -fsized-deallocation  -UNDEBUG -gdwarf-aranges -pipe -mssse3 -msse4.1 -msse4.2 -mpclmul -mpopcnt -fasynchronous-unwind-tables -falign-functions=32 -mbranches-within-32B-boundaries -fdiagnostics-absolute-paths -fstrict-vtable-pointers -fexperimental-new-pass-manager -Wall -Wextra -Weverything -Wpedantic -Wno-zero -length-array -Wno-c++98-compat-pedantic -Wno-c++98-compat -Wno-c++20-compat -Wno-conversion -Wno-ctad-maybe-unsupported -Wno-disabled-macro-expansion -Wno-documentation-unknown-command -Wno-double-promotion -Wno-exit-time-destructors -Wno-float-equal -Wno-global-constructors -Wno-missing-prototypes -Wno-missing-variable-declarations -Wno-padded -Wno-switch-enum -Wno-undefined-func-template -Wno-unused-template -Wno-vla -Wno-weak-template-vtables -Wno-weak-vtables -Wno-thread-safety-negative -g -O0 -g -gdwarf-4 -fno-inline  -D_LIBCPP_DEBUG=0   -D OS_LINUX -I/ch/base -I/ch/contrib/magic_enum/include -include /ch/src/Core/iostream_debug_helpers.h -Werror -nostdinc++ -std=gnu++2a -MD -MT src/CMakeFiles/dbms.dir/Processors/Formats/Impl/MsgPackRowInputFormat.cpp.o -MF src/CMakeFiles/dbms.dir/Processors/Formats/Impl/MsgPackRowInputFormat.cpp.o.d -o src/CMakeFiles/dbms.dir/Processors/Formats/Impl/MsgPackRowInputFormat.cpp.o -c /ch/src/Processors/Formats/Impl/MsgPackRowInputFormat.cpp

    /ch/contrib/msgpack-c/include/msgpack/v1/detail/cpp11_zone.hpp:195:9: error: Attempt to free released memory [clang-analyzer-cplusplus.NewDelete,-warnings-as-errors]
            ::free(p);
            ^
    /ch/src/Processors/Formats/Impl/MsgPackRowInputFormat.cpp:509:5: note: Taking false branch
        if (buf.eof())
        ^
    /ch/src/Processors/Formats/Impl/MsgPackRowInputFormat.cpp:514:24: note: Assuming 'i' is not equal to field 'number_of_columns'
        for (size_t i = 0; i != number_of_columns; ++i)
                           ^~~~~~~~~~~~~~~~~~~~~~
    /ch/src/Processors/Formats/Impl/MsgPackRowInputFormat.cpp:514:5: note: Loop condition is true.  Entering loop body
        for (size_t i = 0; i != number_of_columns; ++i)
        ^
    /ch/src/Processors/Formats/Impl/MsgPackRowInputFormat.cpp:516:30: note: Calling 'MsgPackSchemaReader::readObject'
            auto object_handle = readObject();
                                 ^~~~~~~~~~~~
    /ch/src/Processors/Formats/Impl/MsgPackRowInputFormat.cpp:426:5: note: Taking false branch
        if (buf.eof())
        ^
    /ch/src/Processors/Formats/Impl/MsgPackRowInputFormat.cpp:433:5: note: Loop condition is true.  Entering loop body
        while (need_more_data)
        ^
    /ch/src/Processors/Formats/Impl/MsgPackRowInputFormat.cpp:438:29: note: Calling 'unpack'
                object_handle = msgpack::unpack(buf.position(), buf.buffer().end() - buf.position(), offset);
                                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    /ch/contrib/msgpack-c/include/msgpack/v3/unpack.hpp:52:12: note: Calling 'unpack'
        return msgpack::v3::unpack(data, len, off, referenced, f, user_data, limit);
               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    /ch/contrib/msgpack-c/include/msgpack/v3/unpack.hpp:35:5: note: Control jumps to the 'default' case at line 40
        switch(ret) {
        ^
    /ch/contrib/msgpack-c/include/msgpack/v3/unpack.hpp:41:9: note:  Execution continues on line 43
            break;
            ^
    /ch/contrib/msgpack-c/include/msgpack/v3/unpack.hpp:43:35: note: Calling '~unique_ptr'
        return msgpack::object_handle();
                                      ^
    /ch/contrib/libcxx/include/__memory/unique_ptr.h:269:19: note: Calling 'unique_ptr::reset'
      ~unique_ptr() { reset(); }
                      ^~~~~~~
    /ch/contrib/libcxx/include/__memory/unique_ptr.h:314:9: note: '__tmp' is non-null
        if (__tmp)
            ^~~~~
    /ch/contrib/libcxx/include/__memory/unique_ptr.h:314:5: note: Taking true branch
        if (__tmp)
        ^
    /ch/contrib/libcxx/include/__memory/unique_ptr.h:315:7: note: Calling 'default_delete::operator()'
          __ptr_.second()(__tmp);
          ^~~~~~~~~~~~~~~~~~~~~~
    /ch/contrib/libcxx/include/__memory/unique_ptr.h:54:5: note: Memory is released
        delete __ptr;
        ^~~~~~~~~~~~
    /ch/contrib/libcxx/include/__memory/unique_ptr.h:54:5: note: Calling 'zone::operator delete'
        delete __ptr;
        ^~~~~~~~~~~~
    /ch/contrib/msgpack-c/include/msgpack/v1/detail/cpp11_zone.hpp:195:9: note: Attempt to free released memory
            ::free(p);
            ^~~~~~~~~

  [1]: https://s3.amazonaws.com/clickhouse-builds/41046/9677898b3b234a5ba0371edaf719ea8890d084ff/binary_tidy/build_log.log

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-09-10 21:38:35 +02:00
Azat Khuzhin
4e3135383b Fix clang-tidy warnings (from clang-15) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-09-10 21:38:35 +02:00
Azat Khuzhin
ba4174402a Temporary disable complation of compileAddIntoAggregateStatesFunctions 
Fixes the following use-of-uninitialized-value in llvm [1]:

    ==696==WARNING: MemorySanitizer: use-of-uninitialized-value
        0 0x498141d9 in llvm::ilist_traits<llvm::MachineInstr>::removeNodeFromList(llvm::MachineInstr*) build_docker/../contrib/llvm/llvm/lib/CodeGen/MachineBasicBlock.cpp:155:24
        1 0x498141d9 in llvm::iplist_impl<llvm::simple_ilist<llvm::MachineInstr, llvm::ilist_sentinel_tracking<true>>, llvm::ilist_traits<llvm::MachineInstr>>::remove(llvm::ilist_iterator<llvm::ilist_detail::node_options<llvm::MachineInstr, true, true, void>, false, false>&) build_docker/../contrib/llvm/llvm/include/llvm/ADT/ilist.h:253:11
        2 0x498141d9 in llvm::iplist_impl<llvm::simple_ilist<llvm::MachineInstr, llvm::ilist_sentinel_tracking<true>>, llvm::ilist_traits<llvm::MachineInstr>>::erase(llvm::ilist_iterator<llvm::ilist_detail::node_options<llvm::MachineInstr, true, true, void>, false, false>) build_docker/../contrib/llvm/llvm/include/llvm/ADT/ilist.h:268:22
        3 0x498141d9 in llvm::iplist_impl<llvm::simple_ilist<llvm::MachineInstr, llvm::ilist_sentinel_tracking<true>>, llvm::ilist_traits<llvm::MachineInstr>>::erase(llvm::ilist_iterator<llvm::ilist_detail::node_options<llvm::MachineInstr, true, true, void>, false, false>, llvm::ilist_iterator<llvm::ilist_detail::node_options<llvm::MachineInstr, true, true, void>, false, false>) build_docker/../contrib/llvm/llvm/include/llvm/ADT/ilist.h:305:15
        4 0x498141d9 in llvm::iplist_impl<llvm::simple_ilist<llvm::MachineInstr, llvm::ilist_sentinel_tracking<true>>, llvm::ilist_traits<llvm::MachineInstr>>::clear() build_docker/../contrib/llvm/llvm/include/llvm/ADT/ilist.h:309:18
        5 0x498141d9 in llvm::iplist_impl<llvm::simple_ilist<llvm::MachineInstr, llvm::ilist_sentinel_tracking<true>>, llvm::ilist_traits<llvm::MachineInstr>>::~iplist_impl() build_docker/../contrib/llvm/llvm/include/llvm/ADT/ilist.h:210:20
        6 0x498145ff in llvm::MachineBasicBlock::~MachineBasicBlock() build_docker/../contrib/llvm/llvm/lib/CodeGen/MachineBasicBlock.cpp:56:1
        7 0x49969c1f in llvm::MachineFunction::DeleteMachineBasicBlock(llvm::MachineBasicBlock*) build_docker/../contrib/llvm/llvm/lib/CodeGen/MachineFunction.cpp:426:8
        8 0x49969c1f in llvm::ilist_alloc_traits<llvm::MachineBasicBlock>::deleteNode(llvm::MachineBasicBlock*) build_docker/../contrib/llvm/llvm/lib/CodeGen/MachineFunction.cpp:127:21
        9 0x4983f4d6 in llvm::iplist_impl<llvm::simple_ilist<llvm::MachineBasicBlock>, llvm::ilist_traits<llvm::MachineBasicBlock>>::erase(llvm::ilist_iterator<llvm::ilist_detail::node_options<llvm::MachineBasicBlock, false, false, void>, false, false>) build_docker/../contrib/llvm/llvm/include/llvm/ADT/ilist.h:268:11
        10 0x4983f4d6 in llvm::iplist_impl<llvm::simple_ilist<llvm::MachineBasicBlock>, llvm::ilist_traits<llvm::MachineBasicBlock>>::erase(llvm::MachineBasicBlock*) build_docker/../contrib/llvm/llvm/include/llvm/ADT/ilist.h:272:39
        11 0x4983f4d6 in llvm::MachineFunction::erase(llvm::MachineBasicBlock*) build_docker/../contrib/llvm/llvm/include/llvm/CodeGen/MachineFunction.h:767:53
        12 0x4983f4d6 in llvm::MachineBasicBlock::eraseFromParent() build_docker/../contrib/llvm/llvm/lib/CodeGen/MachineBasicBlock.cpp:1317:16
        13 0x4a0c9a27 in llvm::TailDuplicator::removeDeadBlock(llvm::MachineBasicBlock*, llvm::function_ref<void (llvm::MachineBasicBlock*)>*) build_docker/../contrib/llvm/llvm/lib/CodeGen/TailDuplicator.cpp:1051:8
        14 0x4a0c1e41 in llvm::TailDuplicator::tailDuplicateAndUpdate(bool, llvm::MachineBasicBlock*, llvm::MachineBasicBlock*, llvm::SmallVectorImpl<llvm::MachineBasicBlock*>*, llvm::function_ref<void (llvm::MachineBasicBlock*)>*, llvm::SmallVectorImpl<llvm::MachineBasicBlock*>*) build_docker/../contrib/llvm/llvm/lib/CodeGen/TailDuplicator.cpp:189:5
        15 0x4a0ca16e in llvm::TailDuplicator::tailDuplicateBlocks() build_docker/../contrib/llvm/llvm/lib/CodeGen/TailDuplicator.cpp:288:19
        16 0x4a0be9f9 in (anonymous namespace)::TailDuplicateBase::runOnMachineFunction(llvm::MachineFunction&) build_docker/../contrib/llvm/llvm/lib/CodeGen/TailDuplication.cpp:98:21
        17 0x499a2777 in llvm::MachineFunctionPass::runOnFunction(llvm::Function&) build_docker/../contrib/llvm/llvm/lib/CodeGen/MachineFunctionPass.cpp:72:13
        18 0x4dbba34d in llvm::FPPassManager::runOnFunction(llvm::Function&) build_docker/../contrib/llvm/llvm/lib/IR/LegacyPassManager.cpp:1435:27
        19 0x4dbe3761 in llvm::FPPassManager::runOnModule(llvm::Module&) build_docker/../contrib/llvm/llvm/lib/IR/LegacyPassManager.cpp:1481:16
        20 0x4dbbebbb in (anonymous namespace)::MPPassManager::runOnModule(llvm::Module&) build_docker/../contrib/llvm/llvm/lib/IR/LegacyPassManager.cpp:1550:27
        21 0x4dbbebbb in llvm::legacy::PassManagerImpl::run(llvm::Module&) build_docker/../contrib/llvm/llvm/lib/IR/LegacyPassManager.cpp:541:44
        22 0x4dbe454b in llvm::legacy::PassManager::run(llvm::Module&) build_docker/../contrib/llvm/llvm/lib/IR/LegacyPassManager.cpp:1677:14
        23 0x414405df in DB::JITCompiler::compile(llvm::Module&) build_docker/../src/Interpreters/JIT/CHJIT.cpp:78:22
        24 0x4143bb7d in DB::CHJIT::compileModule(std::__1::unique_ptr<llvm::Module, std::__1::default_delete<llvm::Module>>) build_docker/../src/Interpreters/JIT/CHJIT.cpp:378:29
        25 0x4143aded in DB::CHJIT::compileModule(std::__1::function<void (llvm::Module&)>) build_docker/../src/Interpreters/JIT/CHJIT.cpp:359:24
        26 0x4147b25e in DB::compileAggregateFunctions(DB::CHJIT&, std::__1::vector<DB::AggregateFunctionWithOffset, std::__1::allocator<DB::AggregateFunctionWithOffset>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>) build_docker/../src/Interpreters/JIT/compileFunction.cpp:738:32
        27 0x3de0a23a in DB::Aggregator::compileAggregateFunctionsIfNeeded()::$_0::operator()() const build_docker/../src/Interpreters/Aggregator.cpp:661:49
        28 0x3de0a23a in std::__1::pair<std::__1::shared_ptr<DB::CompiledExpressionCacheEntry>, bool> DB::CacheBase<wide::integer<128ul, unsigned int>, DB::CompiledExpressionCacheEntry, UInt128Hash, DB::CompiledFunctionWeightFunction>::getOrSet<DB::Aggregator::compileAggregateFunctionsIfNeeded()::$_0>(wide::integer<128ul, unsigned int> const&, DB::Aggregator::compileAggregateFunctionsIfNeeded()::$_0&&) build_docker/../src/Common/CacheBase.h:125:24
        29 0x3de0a23a in DB::Aggregator::compileAggregateFunctionsIfNeeded() build_docker/../src/Interpreters/Aggregator.cpp:657:70

      Memory was marked as uninitialized
        0 0xb988ded in __sanitizer_dtor_callback (/usr/bin/clickhouse+0xb988ded) (BuildId: c4a880b742797a1f37bc4f5ed869f055cc86486b)
        1 0x498145da in llvm::MachineBasicBlock::~MachineBasicBlock() build_docker/../contrib/llvm/llvm/lib/CodeGen/MachineBasicBlock.cpp:56:1
        2 0x49969c1f in llvm::MachineFunction::DeleteMachineBasicBlock(llvm::MachineBasicBlock*) build_docker/../contrib/llvm/llvm/lib/CodeGen/MachineFunction.cpp:426:8
        3 0x49969c1f in llvm::ilist_alloc_traits<llvm::MachineBasicBlock>::deleteNode(llvm::MachineBasicBlock*) build_docker/../contrib/llvm/llvm/lib/CodeGen/MachineFunction.cpp:127:21
        4 0x4983f4d6 in llvm::iplist_impl<llvm::simple_ilist<llvm::MachineBasicBlock>, llvm::ilist_traits<llvm::MachineBasicBlock>>::erase(llvm::ilist_iterator<llvm::ilist_detail::node_options<llvm::MachineBasicBlock, false, false, void>, false, false>) build_docker/../contrib/llvm/llvm/include/llvm/ADT/ilist.h:268:11
        5 0x4983f4d6 in llvm::iplist_impl<llvm::simple_ilist<llvm::MachineBasicBlock>, llvm::ilist_traits<llvm::MachineBasicBlock>>::erase(llvm::MachineBasicBlock*) build_docker/../contrib/llvm/llvm/include/llvm/ADT/ilist.h:272:39
        6 0x4983f4d6 in llvm::MachineFunction::erase(llvm::MachineBasicBlock*) build_docker/../contrib/llvm/llvm/include/llvm/CodeGen/MachineFunction.h:767:53
        7 0x4983f4d6 in llvm::MachineBasicBlock::eraseFromParent() build_docker/../contrib/llvm/llvm/lib/CodeGen/MachineBasicBlock.cpp:1317:16
        8 0x4a0c9a27 in llvm::TailDuplicator::removeDeadBlock(llvm::MachineBasicBlock*, llvm::function_ref<void (llvm::MachineBasicBlock*)>*) build_docker/../contrib/llvm/llvm/lib/CodeGen/TailDuplicator.cpp:1051:8
        9 0x4a0c1e41 in llvm::TailDuplicator::tailDuplicateAndUpdate(bool, llvm::MachineBasicBlock*, llvm::MachineBasicBlock*, llvm::SmallVectorImpl<llvm::MachineBasicBlock*>*, llvm::function_ref<void (llvm::MachineBasicBlock*)>*, llvm::SmallVectorImpl<llvm::MachineBasicBlock*>*) build_docker/../contrib/llvm/llvm/lib/CodeGen/TailDuplicator.cpp:189:5
        10 0x4a0ca16e in llvm::TailDuplicator::tailDuplicateBlocks() build_docker/../contrib/llvm/llvm/lib/CodeGen/TailDuplicator.cpp:288:19
        11 0x4a0be9f9 in (anonymous namespace)::TailDuplicateBase::runOnMachineFunction(llvm::MachineFunction&) build_docker/../contrib/llvm/llvm/lib/CodeGen/TailDuplication.cpp:98:21
        12 0x499a2777 in llvm::MachineFunctionPass::runOnFunction(llvm::Function&) build_docker/../contrib/llvm/llvm/lib/CodeGen/MachineFunctionPass.cpp:72:13
        13 0x4dbba34d in llvm::FPPassManager::runOnFunction(llvm::Function&) build_docker/../contrib/llvm/llvm/lib/IR/LegacyPassManager.cpp:1435:27
        14 0x4dbe3761 in llvm::FPPassManager::runOnModule(llvm::Module&) build_docker/../contrib/llvm/llvm/lib/IR/LegacyPassManager.cpp:1481:16
        15 0x4dbbebbb in (anonymous namespace)::MPPassManager::runOnModule(llvm::Module&) build_docker/../contrib/llvm/llvm/lib/IR/LegacyPassManager.cpp:1550:27
        16 0x4dbbebbb in llvm::legacy::PassManagerImpl::run(llvm::Module&) build_docker/../contrib/llvm/llvm/lib/IR/LegacyPassManager.cpp:541:44
        17 0x4dbe454b in llvm::legacy::PassManager::run(llvm::Module&) build_docker/../contrib/llvm/llvm/lib/IR/LegacyPassManager.cpp:1677:14
        18 0x414405df in DB::JITCompiler::compile(llvm::Module&) build_docker/../src/Interpreters/JIT/CHJIT.cpp:78:22
        19 0x4143bb7d in DB::CHJIT::compileModule(std::__1::unique_ptr<llvm::Module, std::__1::default_delete<llvm::Module>>) build_docker/../src/Interpreters/JIT/CHJIT.cpp:378:29
        20 0x4143aded in DB::CHJIT::compileModule(std::__1::function<void (llvm::Module&)>) build_docker/../src/Interpreters/JIT/CHJIT.cpp:359:24
        21 0x4147b25e in DB::compileAggregateFunctions(DB::CHJIT&, std::__1::vector<DB::AggregateFunctionWithOffset, std::__1::allocator<DB::AggregateFunctionWithOffset>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>) build_docker/../src/Interpreters/JIT/compileFunction.cpp:738:32
        22 0x3de0a23a in DB::Aggregator::compileAggregateFunctionsIfNeeded()::$_0::operator()() const build_docker/../src/Interpreters/Aggregator.cpp:661:49
        23 0x3de0a23a in std::__1::pair<std::__1::shared_ptr<DB::CompiledExpressionCacheEntry>, bool> DB::CacheBase<wide::integer<128ul, unsigned int>, DB::CompiledExpressionCacheEntry, UInt128Hash, DB::CompiledFunctionWeightFunction>::getOrSet<DB::Aggregator::compileAggregateFunctionsIfNeeded()::$_0>(wide::integer<128ul, unsigned int> const&, DB::Aggregator::compileAggregateFunctionsIfNeeded()::$_0&&) build_docker/../src/Common/CacheBase.h:125:24
        24 0x3de0a23a in DB::Aggregator::compileAggregateFunctionsIfNeeded() build_docker/../src/Interpreters/Aggregator.cpp:657:70

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/41046/490a2c75610c4bc3191d55226f8454b3c3d3919a/stateful_tests__msan_.html

Note, that it is safe to do, but only for this method, since it had been
disabled anyway, back in #27574, and I guess this MSan report may be
related.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-09-10 21:38:35 +02:00
Azat Khuzhin
e0a8f19cfb Fix -Wzero-as-null-pointer-constant in MemoryStatisticsOS::get for FreeBSD 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-09-10 21:38:35 +02:00
Azat Khuzhin
e8d7403a38 Suppress warning in FormatFactory::getFormatFromFileDescriptor() for FreeBSD 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-09-10 21:38:35 +02:00
Alexey Milovidov
871ee15a9c
Merge pull request #40953 from ClibMouse/sensitive_data_masker_unittest_issue 
Rearrange sensitive data masker unit test
 2022-09-10 20:03:30 +03:00
Alexey Milovidov
89bae332a0
Merge pull request #41170 from ClickHouse/fix-weird-lowcardinality 
Fix something terribly wrong in LowCardinality
 2022-09-10 18:08:35 +03:00
Kruglov Pavel
2ca5846595
Merge pull request #41159 from azat/load-balancing-fix 
Fix stack-use-after-return in GetPriorityForLoadBalancing::getPriorityFunc()
 2022-09-10 15:11:31 +02:00
Alexey Milovidov
61278c81e4 Merge branch 'master' into remove-useless-line 2022-09-10 05:07:10 +02:00
Alexey Milovidov
fd235919aa Remove some methods 2022-09-10 05:04:40 +02:00
Suzy Wang
d60340eb40
Merge branch 'master' into sensitive_data_masker_unittest_issue 2022-09-09 22:59:50 -04:00
Alexey Milovidov
fa62c7e982 Fix half of trash 2022-09-10 04:08:16 +02:00
Alexey Milovidov
daff201566 Fix something terribly wrong in LowCardinality 2022-09-10 03:28:37 +02:00
Alexey Milovidov
4301b362ab Fix strange code in DatabaseReplicated 2022-09-10 02:34:37 +02:00
Igor Nikonov
726639484a
Revert "Query plan optimization setting: read in window order" 2022-09-09 22:15:38 +02:00
avogar
d8cbd51333 Support relative path in Location header after http redirect 2022-09-09 19:26:14 +00:00
Azat Khuzhin
8c5583d7a5 Fix stack-use-after-return in GetPriorityForLoadBalancing::getPriorityFunc() 
clang-15 reports [1]:

<details>

<summary>ASan report</summary>

```
    ==1==ERROR: AddressSanitizer: stack-use-after-return on address 0x7f1d04c4eb20 at pc 0x000031c4803c bp 0x7f1d05e19a00 sp 0x7f1d05e199f8
    READ of size 8 at 0x7f1d04c4eb20 thread T200 (QueryPullPipeEx)
        #0 0x31c4803b in DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3::operator()(unsigned long) const build_docker/../src/Common/GetPriorityForLoadBalancing.cpp:42:40
        #1 0x31c4803b in decltype(static_cast<DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3&>(fp)(static_cast<unsigned long>(fp0))) std::__1::__invoke<DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3&, unsigned long>(DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3&, unsigned long&&) build_docker/../contrib/libcxx/include/type_traits:3640:23
        #2 0x31c4803b in unsigned long std::__1::__invoke_void_return_wrapper<unsigned long, false>::__call<DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3&, unsigned long>(DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3&, unsigned long&&) build_docker/../contrib/libcxx/include/__functional/invoke.h:30:16
        #3 0x31c4803b in std::__1::__function::__default_alloc_func<DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3, unsigned long (unsigned long)>::operator()(unsigned long&&) build_docker/../contrib/libcxx/include/__functional/function.h:230:12
        #4 0x31c4803b in unsigned long std::__1::__function::__policy_invoker<unsigned long (unsigned long)>::__call_impl<std::__1::__function::__default_alloc_func<DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3, unsigned long (unsigned long)>>(std::__1::__function::__policy_storage const*, unsigned long) build_docker/../contrib/libcxx/include/__functional/function.h:711:16
        #5 0x31c38b07 in std::__1::__function::__policy_func<unsigned long (unsigned long)>::operator()(unsigned long&&) const build_docker/../contrib/libcxx/include/__functional/function.h:843:16
        #6 0x31c38b07 in std::__1::function<unsigned long (unsigned long)>::operator()(unsigned long) const build_docker/../contrib/libcxx/include/__functional/function.h:1184:12
        #7 0x31c38b07 in PoolWithFailoverBase<DB::IConnectionPool>::getShuffledPools(unsigned long, std::__1::function<unsigned long (unsigned long)> const&) build_docker/../src/Common/PoolWithFailoverBase.h:174:39

      This frame has 2 object(s):
        [32, 40) 'pool_size.addr' <== Memory access at offset 32 is inside this variable
        [64, 88) 'ref.tmp' (line 18)
```

</details>

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/41046/adea92f847373d1fcfd733d8979c63024f9b80bf/integration_tests__asan__[1/3].html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-09-09 20:06:48 +02:00
Denny Crane
be90cecbd2 Revert "Revert "Fix trivial count optimization with array join"" 
This reverts commit 17de7b2876.
 2022-09-09 14:56:38 -03:00
Azat Khuzhin
763bb18f98 Fix SIGSEGV in SortedBlocksWriter in case of empty block 
CI found one issue [1].

Here is the stack trace for invalid read:

<details>

<summary>stack trace</summary>

```
    0: DB::TemporaryFileLazySource::TemporaryFileLazySource(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Block const&) [inlined] std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::__is_long(this="") const at string:1445:22
    1: DB::TemporaryFileLazySource::TemporaryFileLazySource(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Block const&) [inlined] std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::basic_string(this="", __str="") at string:1927
    2: DB::TemporaryFileLazySource::TemporaryFileLazySource(this=0x00007f3aec105f58, path_="", header_=0x00007f38ffd93b40) at TemporaryFileLazySource.cpp:11
    3: DB::SortedBlocksWriter::streamFromFile(std::__1::unique_ptr<Poco::TemporaryFile, std::__1::default_delete<Poco::TemporaryFile> > const&) const [inlined] DB::TemporaryFileLazySource* std::__1::construct_at<DB::TemporaryFileLazySource, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Block, DB::TemporaryFileLazySource*>(__args=0x00007f38ffd91560) at construct_at.h:38:50
    4: DB::SortedBlocksWriter::streamFromFile(std::__1::unique_ptr<Poco::TemporaryFile, std::__1::default_delete<Poco::TemporaryFile> > const&) const [inlined] void std::__1::allocator_traits<std::__1::allocator<DB::TemporaryFileLazySource> >::construct<DB::TemporaryFileLazySource, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Block, void, void>(__args=0x00007f38ffd91560) at allocator_traits.h:298
    5: DB::SortedBlocksWriter::streamFromFile(std::__1::unique_ptr<Poco::TemporaryFile, std::__1::default_delete<Poco::TemporaryFile> > const&) const [inlined] std::__1::__shared_ptr_emplace<DB::TemporaryFileLazySource, std::__1::allocator<DB::TemporaryFileLazySource> >::__shared_ptr_emplace<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Block>(this=0x00007f3aec105f40, __args=0x00007f38ffd91560) at shared_ptr.h:293
    6: DB::SortedBlocksWriter::streamFromFile(std::__1::unique_ptr<Poco::TemporaryFile, std::__1::default_delete<Poco::TemporaryFile> > const&) const [inlined] std::__1::shared_ptr<DB::TemporaryFileLazySource> std::__1::allocate_shared<DB::TemporaryFileLazySource, std::__1::allocator<DB::TemporaryFileLazySource>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Block, void>(__args=<unavailable>, __args=<unavailable>) at shared_ptr.h:954
    7: DB::SortedBlocksWriter::streamFromFile(std::__1::unique_ptr<Poco::TemporaryFile, std::__1::default_delete<Poco::TemporaryFile> > const&) const [inlined] std::__1::shared_ptr<DB::TemporaryFileLazySource> std::__1::make_shared<DB::TemporaryFileLazySource, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Block, void>(__args=<unavailable>, __args=<unavailable>) at shared_ptr.h:963
    8: DB::SortedBlocksWriter::streamFromFile(this=<unavailable>, file=<unavailable>) const at SortedBlocksWriter.cpp:238
    9: DB::SortedBlocksWriter::premerge(this=<unavailable>) at SortedBlocksWriter.cpp:209:32
```

</details>

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/41046/adea92f847373d1fcfd733d8979c63024f9b80bf/stress_test__asan_.html

So the problem here is that there was empty unique_ptr<> reference to
temporary file, because of empty block that accepted by
SortedBlocksWriter::insert(), but insert() is not a problem the problem
is premerge() that steals blocks from insert() and do not have check
that there are some rows. However this check exists in
SortedBlocksWriter::flush(), and in that case temporary file is not
created.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-09-09 19:52:38 +02:00
Azat Khuzhin
57146c9361 Fix typos in SortedBlocksWriter 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-09-09 19:52:38 +02:00
Anton Popov
a1b5582a7c
Merge pull request #37152 from CurtizJ/fix-filling-of-missed-nested 
Fix filling of missed Nested columns with multiple levels
 2022-09-09 19:29:01 +02:00
Alexander Tokmakov
9bcec2d49a stop background ops before converting db engine 2022-09-09 18:27:19 +02:00
zhenjial
51badf3d0a fix 2022-09-10 00:08:50 +08:00
Dmitry Novik
64c4023737
Merge branch 'master' into grouping-comp 2022-09-09 18:05:01 +02:00
Dmitry Novik
877776e569
Update src/Functions/grouping.h 2022-09-09 18:04:29 +02:00