Commit Graph

7594 Commits

Author SHA1 Message Date
Vladimir C
b6b5932efc
Merge pull request #41223 from ClickHouse/vdimir/exponential_rewrite_21557 2022-09-19 10:07:24 +02:00
Duc Canh Le
61d6fae14b add settings allow_suspicious_fixed_string_types 2022-09-19 15:00:04 +08:00
Alexey Milovidov
730655d4fd Fix 8/9 of trash 2022-09-19 08:53:20 +02:00
Alexey Milovidov
ab4db2d0c4 Fix 5/6 of trash 2022-09-19 08:50:53 +02:00
Alexey Milovidov
3c0961c48b
Merge branch 'master' into fix-map-combinator 2022-09-19 09:32:16 +03:00
Alexey Milovidov
b01f5bdca8
Merge pull request #39354 from ClickHouse/window-order-by
Respect ORDER BY clause in window functions
2022-09-19 09:22:37 +03:00
Frank Chen
b056bc1021 Add span for executeDDLQueryOnCluster
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
2022-09-19 13:43:40 +08:00
Yong Wang
37ae7a8cca Kusto-phase1 : apply parser comments to kusto, remove unused variable 2022-09-18 20:25:27 -07:00
Frank Chen
06ae2fb2b5 Remove assertEOF to improve the compability
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
2022-09-19 11:20:58 +08:00
Yong Wang
4e4a04c08b
Merge branch 'ClickHouse:master' into Kusto-phase1 2022-09-18 21:40:30 -04:00
Vitaly Baranov
5365b105cc Add SYSTEM RELOAD USERS command. 2022-09-18 12:44:00 +02:00
Alexey Milovidov
c41e37fa8e Code simplification 2022-09-18 08:58:32 +02:00
Alexey Milovidov
3c43d6516b Fix style 2022-09-18 08:56:52 +02:00
Alexey Milovidov
791de6592b Remove trash from Field 2022-09-18 05:16:08 +02:00
kssenii
51745dcb43 Fix assertion in write-through cache 2022-09-17 20:20:32 +02:00
Anton Popov
07cfb0cba4
Merge branch 'master' into dynamic-columns-21 2022-09-17 19:02:58 +02:00
Igor Nikonov
475443c10c
Merge pull request #41257 from ClickHouse/igor/mixed_improvements_along_the_way
Tiny improvements along the code
2022-09-17 10:43:46 +02:00
Alexey Milovidov
c09604bef2
Merge pull request #41363 from kitaisreal/small-style-fixes
Small style fixes
2022-09-17 00:19:00 +03:00
avogar
c9a9ef5b7e Fix comments 2022-09-16 16:12:30 +00:00
Alexander Tokmakov
65474830ac make tryEnqueueReplicatedDDL virtual 2022-09-16 16:25:32 +02:00
Robert Schulze
b32b02d844
Merge pull request #40897 from ClickHouse/catboost-bridge-resurrected
Move CatBoost evaluation into clickhouse-library-bridge
2022-09-16 13:12:09 +02:00
Alexey Milovidov
da01982652
Merge pull request #41046 from azat/build/llvm-15
Switch to llvm/clang 15
2022-09-16 07:31:06 +03:00
Alexander Tokmakov
03c193ccca fix ON CLUSTER with Replicated database cluster 2022-09-15 21:15:57 +02:00
Alexander Tokmakov
32391f737a Merge branch 'master' into fix_41209 2022-09-15 20:27:51 +02:00
Alexander Tokmakov
b26cc5a1ba
Merge pull request #41296 from young-scott/fix_issuse_41096
fix issue #41096
2022-09-15 15:44:05 +03:00
Alexander Tokmakov
77c4ea9647 fix 2022-09-15 14:34:31 +02:00
Azat Khuzhin
5cc5ca22de Fix possible deadlock with async_socket_for_remote/use_hedged_requests and parallel KILL
Right now it is possible to call QueryStatus::addPipelineExecutor() when
the executors_mutex already acquired, it is possible when the query was
cancelled via KILL QUERY.

Here I will show some traces from debugger from a real example, where
tons of ProcessList::insert() got deadlocked.

Let's look at the lock owner for one of the threads that was deadlocked
in ProcessList::insert():

    (gdb) p *mutex
    $2 = {
      __data = {
        __owner = 46899,
      },
    }

And now let's see the stack trace of the 46899:

    #0  __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:103
    #1  0x00007fb65569b714 in __GI___pthread_mutex_lock (mutex=0x7fb4a9d15298) at ../nptl/pthread_mutex_lock.c:80
    #2  0x000000001b6edd91 in pthread_mutex_lock (arg=0x7fb4a9d15298) at ../src/Common/ThreadFuzzer.cpp:317
    #3  std::__1::__libcpp_mutex_lock (__m=0x7fb4a9d15298) at ../contrib/libcxx/include/__threading_support:303
    #4  std::__1::mutex::lock (this=0x7fb4a9d15298) at ../contrib/libcxx/src/mutex.cpp:33
    #5  0x0000000014c7ae63 in std::__1::lock_guard<std::__1::mutex>::lock_guard (__m=..., this=<optimized out>) at ../contrib/libcxx/include/__mutex_base:91
    #6  DB::QueryStatus::addPipelineExecutor (this=0x7fb4a9d14f90, e=0x80) at ../src/Interpreters/ProcessList.cpp:372
    #7  0x0000000015bee4a7 in DB::PipelineExecutor::PipelineExecutor (this=0x7fb4b1e53618, processors=..., elem=<optimized out>) at ../src/Processors/Executors/PipelineExecutor.cpp:54
    #12 std::__1::make_shared<DB::PipelineExecutor, std::__1::vector<std::__1::shared_ptr<DB::IProcessor>, std::__1::allocator<std::__1::shared_ptr<DB::IProcessor> > >&, DB::QueryStatus*&, void> (__args=@0x7fb63095b9b0: 0x7fb4a9d14f90, __args=@0x7fb63095b9b0: 0x7fb4a9d14f90) at ../contrib/libcxx/include/__memory/shared_ptr.h:963
    #13 DB::QueryPipelineBuilder::execute (this=0x7fb63095b8b0) at ../src/QueryPipeline/QueryPipelineBuilder.cpp:552
    #14 0x00000000158c6c27 in DB::Connection::sendExternalTablesData (this=0x7fb6545e9d98, data=...) at ../src/Client/Connection.cpp:797
    #27 0x0000000014043a81 in DB::RemoteQueryExecutorRoutine::operator() (this=0x7fb63095bf20, sink=...) at ../src/QueryPipeline/RemoteQueryExecutorReadContext.cpp:46
    #32 0x000000000a16dd4f in make_fcontext () at ../contrib/boost/libs/context/src/asm/make_x86_64_sysv_elf_gas.S:71

And also in the logs you can see very strange things for this thread:

    2022.09.13 14:14:51.228979 [ 51145 ] {1712D4E914EC7C99} <Debug> Connection (localhost:9000): Sent data for 1 external tables, total 11 rows in 0.00046389 sec., 23688 rows/sec., 3.84 KiB (8.07 MiB/sec.), compressed 1.1070121092649958 times to 3.47 KiB (7.29 MiB/sec.)
    ...
    2022.09.13 14:14:51.719402 [ 46899 ] {7c90ffa4-1dc8-42fd-938c-4e307c244394} <Debug> executeQuery: (from 10.101.15.181:42478) KILL QUERY WHERE query_id = '1712D4E914EC7C99' (stage: Complete)
    2022.09.13 14:14:51.719488 [ 46899 ] {7c90ffa4-1dc8-42fd-938c-4e307c244394} <Debug> executeQuery: (internal) SELECT query_id, user, query FROM system.processes WHERE query_id = '1712D4E914EC7C99' (stage: Complete)
    2022.09.13 14:14:51.719754 [ 46899 ] {7c90ffa4-1dc8-42fd-938c-4e307c244394} <Trace> ContextAccess (default): Access granted: SELECT(user, query_id, query) ON system.processes
    2022.09.13 14:14:51.720544 [ 46899 ] {7c90ffa4-1dc8-42fd-938c-4e307c244394} <Trace> InterpreterSelectQuery: FetchColumns -> Complete
    2022.09.13 14:14:53.228964 [ 46899 ] {7c90ffa4-1dc8-42fd-938c-4e307c244394} <Debug> Connection (localhost:9000): Sent data for 2 scalars, total 2 rows in 2.6838e-05 sec., 73461 rows/sec., 68.00 B (2.38 MiB/sec.), compressed 0.4594594594594595 times to 148.00 B (5.16 MiB/sec.)

How is this possible? The answer is fibers and query cancellation
routine. During cancellation of async queries it going into fibers again
and try to do this gracefully. However because of this during canceling
query it may call QueryStatus::addPipelineExecutor() from
QueryStatus::cancelQuery().

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-09-15 14:41:27 +03:00
Maksim Kita
276d3628e9 Small style fixes 2022-09-15 13:05:18 +02:00
Frank Chen
f576d8191c Merge branch 'master' into on_cluster_dll 2022-09-15 18:05:36 +08:00
Frank Chen
ac848727e5 Read tracing context in right pos 2022-09-15 18:00:08 +08:00
Nikolay Degterinsky
72d8a04ee6 Add setting 'allow_implicit_no_password' 2022-09-15 08:07:42 +00:00
Robert Schulze
b64af5989e
Merge branch 'master' into opentelemetry_stress 2022-09-15 09:43:11 +02:00
Frank Chen
a4ef0c0281 Set up tracing context for DDLWorker 2022-09-15 14:57:00 +08:00
Frank Chen
490089cc4f Copy tracing context from current thread to DDLLogEntry as parent context 2022-09-15 14:48:24 +08:00
Frank Chen
52224875e2 Serialize tracing context to DDL log entry 2022-09-15 14:47:43 +08:00
Anton Popov
66667737f0 fix writing of empty columns of type Object 2022-09-15 03:38:22 +00:00
avogar
f1a0501eb2 Fix memory leaks and segfaults in combinators 2022-09-14 18:01:49 +00:00
Alexander Tokmakov
fc73134145 fix race between DROP and short form of ATTACH 2022-09-14 18:03:36 +02:00
Kruglov Pavel
3396ff6c3a
Merge pull request #40516 from zjial/record_errors_for_import_by_csv
Record errors while reading text formats (CSV, TSV).
2022-09-14 12:52:32 +02:00
young scott
b6888022cd
Merge branch 'master' into fix_issuse_41096 2022-09-14 17:38:22 +08:00
young scott
bc111b5680 fix issuse#41096 2022-09-14 09:15:40 +00:00
Alexander Tokmakov
cf7d928554
Update PredicateRewriteVisitor.cpp 2022-09-13 22:38:55 +03:00
Alexander Tokmakov
0175f3d0d7 fix WHERE pushdown when subquery has OFFSET 2022-09-13 20:44:45 +02:00
Vladimir C
f0e4088c84
Merge pull request #41154 from azat/fix-sorted-block-writer 2022-09-13 19:14:45 +02:00
Dmitry Novik
7e3fb0a681
Merge pull request #40762 from ClickHouse/grouping-comp
Fix GROUPING function SQL compatibility
2022-09-13 17:03:06 +02:00
kssenii
52ef3758c4 Merge remote-tracking branch 'upstream/master' into fix-thread-status 2022-09-13 16:34:31 +02:00
Kruglov Pavel
faa62f96ae
Merge branch 'master' into new-json-formats 2022-09-13 14:41:13 +02:00
Antonio Andelic
baf7255cff
Merge pull request #39976 from ClickHouse/keeper-storage
KeeperMap storage engine
2022-09-13 14:15:09 +02:00
Anton Popov
1989380485
Merge pull request #41122 from lthaooo/optimize_compact_part_mutation
optimize compact part mutation
2022-09-13 14:10:21 +02:00
Igor Nikonov
d8acb0e7ad Tiny improvements along the code
- less allocations (potentially)
2022-09-13 12:06:56 +00:00
Alexander Tokmakov
4d146b05a9
Merge pull request #38262 from PolyProgrammist/fix-ordinary-system-unfreeze
Fix SYSTEM UNFREEZE for ordinary database
2022-09-13 14:55:11 +03:00
Alexander Tokmakov
2845494624
Merge pull request #41120 from azat/async-insert-queue-term
Terminate AsynchronousInsertQueue w/o waiting for timeout
2022-09-13 14:52:50 +03:00
Kseniia Sumarokova
b2c9c04c7b
Merge pull request #40821 from kssenii/improve-marks-cache-loading
Allow to load marks in threadpool in advance
2022-09-13 12:31:58 +02:00
Frank Chen
d3265150c0 Make sure span is finished in the onFinish callback
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
2022-09-13 18:28:21 +08:00
Kseniia Sumarokova
3dd28bb79a
Merge pull request #41042 from kssenii/refactor-cache-for-async-download
Cache refactoring for async download
2022-09-13 12:26:35 +02:00
Robert Schulze
fac1be9700
chore: restore SYSTEM RELOAD MODEL(S) and moniting view SYSTEM.MODELS
- This commit restores statements "SYSTEM RELOAD MODEL(S)" which provide
  a mechanism to update a model explicitly. It also saves potentially
  unnecessary reloads of a model from disk after it's initial load.

  To keep the complexity low, the semantics of "SYSTEM RELOAD MODEL(S)
  was changed from eager to lazy. This means that both statements
  previously immedately reloaded the specified/all models, whereas now
  the statements only trigger an unload and the first call to
  catboostEvaluate() does the actual load.

- Monitoring view SYSTEM.MODELS is also restored but with some obsolete
  fields removed. The view was not documented in the past and for now it
  remains undocumented. The commit is thus not considered a breach of
  ClickHouse's public interface.
2022-09-12 19:33:02 +00:00
Kseniia Sumarokova
5d1e9822f8
Merge branch 'master' into fix-filesystem_cache_log 2022-09-12 20:32:42 +02:00
Alexander Tokmakov
05b4303234
Merge pull request #41146 from ClickHouse/followup_39933
Stop background operations before converting database from Ordinary
2022-09-12 21:20:25 +03:00
Kruglov Pavel
060adfbe93
Merge branch 'master' into new-json-formats 2022-09-12 19:37:46 +02:00
Jose
1880a745b2 Fix String in log entry 2022-09-12 18:53:54 +02:00
vdimir
ba506043ce
Fix exponential query rewrite in case of lots of cross joins with where 2022-09-12 15:05:34 +00:00
Azat Khuzhin
5cb37eb1a0 Terminate AsynchronousInsertQueue w/o waiting for timeout
Previously it simply sleeps for async_insert_busy_timeout_ms delay on
shutdown, which is not great, and produces false-positive hung checks on
CI [1].

  [1]: https://pastila.nl/?0087a100/db80915eac58e67eb05516bf459e5510

Refs: #29390 (cc @tavplubix, @CurtizJ)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-09-12 16:11:54 +02:00
kssenii
413fbb6507 Self-review fixes 2022-09-12 15:46:11 +02:00
Antonio Andelic
1cd43d8e5c Merge branch 'master' into keeper-storage 2022-09-12 13:42:51 +00:00
Anton Popov
76af7efba9
Merge branch 'master' into optimize_compact_part_mutation 2022-09-12 15:35:47 +02:00
Dmitry Novik
c236706703
Merge branch 'master' into grouping-comp 2022-09-12 15:19:51 +02:00
Alexander Tokmakov
ed3c95c28f review fix 2022-09-12 13:53:43 +02:00
Kseniia Sumarokova
d0c058f273
Merge pull request #41132 from Algunenano/views_log_window
Fix query_views_log with Window views
2022-09-12 11:56:22 +02:00
Alexey Milovidov
9c5f9f1815
Merge pull request #41158 from den-crane/bug/trivial_count_optimization_with_array_join
Fix: trivial count optimization with array join. revert revert #39466
2022-09-12 08:56:10 +03:00
Alexey Milovidov
94d6e9ce7d
Merge pull request #41189 from kssenii/fix-positional-args
Fix for positional arguments
2022-09-12 03:21:39 +03:00
Alexey Milovidov
8b5328f7e5 Merge branch 'master' into fix-trash-base58 2022-09-11 20:43:54 +02:00
Alexey Milovidov
3e6c4e3f16 Fix clang-tidy 2022-09-11 20:43:36 +02:00
Denny Crane
0e4e868f29
Merge branch 'master' into bug/trivial_count_optimization_with_array_join 2022-09-11 15:23:10 -03:00
kssenii
9c2bbcf4a6 Fix 2022-09-11 17:38:51 +02:00
kssenii
2f3bfc5411 Fix 2022-09-11 13:54:25 +02:00
kssenii
b7d751b782 Merge remote-tracking branch 'upstream/master' into improve-marks-cache-loading 2022-09-11 13:23:30 +02:00
kssenii
37bdba98b1 Fix 2022-09-11 12:58:04 +02:00
Alexey Milovidov
7f1e7b5967 Merge branch 'master' into fix-half-of-trash 2022-09-11 06:20:47 +02:00
Azat Khuzhin
ba4174402a Temporary disable complation of compileAddIntoAggregateStatesFunctions
Fixes the following use-of-uninitialized-value in llvm [1]:

    ==696==WARNING: MemorySanitizer: use-of-uninitialized-value
        0 0x498141d9 in llvm::ilist_traits<llvm::MachineInstr>::removeNodeFromList(llvm::MachineInstr*) build_docker/../contrib/llvm/llvm/lib/CodeGen/MachineBasicBlock.cpp:155:24
        1 0x498141d9 in llvm::iplist_impl<llvm::simple_ilist<llvm::MachineInstr, llvm::ilist_sentinel_tracking<true>>, llvm::ilist_traits<llvm::MachineInstr>>::remove(llvm::ilist_iterator<llvm::ilist_detail::node_options<llvm::MachineInstr, true, true, void>, false, false>&) build_docker/../contrib/llvm/llvm/include/llvm/ADT/ilist.h:253:11
        2 0x498141d9 in llvm::iplist_impl<llvm::simple_ilist<llvm::MachineInstr, llvm::ilist_sentinel_tracking<true>>, llvm::ilist_traits<llvm::MachineInstr>>::erase(llvm::ilist_iterator<llvm::ilist_detail::node_options<llvm::MachineInstr, true, true, void>, false, false>) build_docker/../contrib/llvm/llvm/include/llvm/ADT/ilist.h:268:22
        3 0x498141d9 in llvm::iplist_impl<llvm::simple_ilist<llvm::MachineInstr, llvm::ilist_sentinel_tracking<true>>, llvm::ilist_traits<llvm::MachineInstr>>::erase(llvm::ilist_iterator<llvm::ilist_detail::node_options<llvm::MachineInstr, true, true, void>, false, false>, llvm::ilist_iterator<llvm::ilist_detail::node_options<llvm::MachineInstr, true, true, void>, false, false>) build_docker/../contrib/llvm/llvm/include/llvm/ADT/ilist.h:305:15
        4 0x498141d9 in llvm::iplist_impl<llvm::simple_ilist<llvm::MachineInstr, llvm::ilist_sentinel_tracking<true>>, llvm::ilist_traits<llvm::MachineInstr>>::clear() build_docker/../contrib/llvm/llvm/include/llvm/ADT/ilist.h:309:18
        5 0x498141d9 in llvm::iplist_impl<llvm::simple_ilist<llvm::MachineInstr, llvm::ilist_sentinel_tracking<true>>, llvm::ilist_traits<llvm::MachineInstr>>::~iplist_impl() build_docker/../contrib/llvm/llvm/include/llvm/ADT/ilist.h:210:20
        6 0x498145ff in llvm::MachineBasicBlock::~MachineBasicBlock() build_docker/../contrib/llvm/llvm/lib/CodeGen/MachineBasicBlock.cpp:56:1
        7 0x49969c1f in llvm::MachineFunction::DeleteMachineBasicBlock(llvm::MachineBasicBlock*) build_docker/../contrib/llvm/llvm/lib/CodeGen/MachineFunction.cpp:426:8
        8 0x49969c1f in llvm::ilist_alloc_traits<llvm::MachineBasicBlock>::deleteNode(llvm::MachineBasicBlock*) build_docker/../contrib/llvm/llvm/lib/CodeGen/MachineFunction.cpp:127:21
        9 0x4983f4d6 in llvm::iplist_impl<llvm::simple_ilist<llvm::MachineBasicBlock>, llvm::ilist_traits<llvm::MachineBasicBlock>>::erase(llvm::ilist_iterator<llvm::ilist_detail::node_options<llvm::MachineBasicBlock, false, false, void>, false, false>) build_docker/../contrib/llvm/llvm/include/llvm/ADT/ilist.h:268:11
        10 0x4983f4d6 in llvm::iplist_impl<llvm::simple_ilist<llvm::MachineBasicBlock>, llvm::ilist_traits<llvm::MachineBasicBlock>>::erase(llvm::MachineBasicBlock*) build_docker/../contrib/llvm/llvm/include/llvm/ADT/ilist.h:272:39
        11 0x4983f4d6 in llvm::MachineFunction::erase(llvm::MachineBasicBlock*) build_docker/../contrib/llvm/llvm/include/llvm/CodeGen/MachineFunction.h:767:53
        12 0x4983f4d6 in llvm::MachineBasicBlock::eraseFromParent() build_docker/../contrib/llvm/llvm/lib/CodeGen/MachineBasicBlock.cpp:1317:16
        13 0x4a0c9a27 in llvm::TailDuplicator::removeDeadBlock(llvm::MachineBasicBlock*, llvm::function_ref<void (llvm::MachineBasicBlock*)>*) build_docker/../contrib/llvm/llvm/lib/CodeGen/TailDuplicator.cpp:1051:8
        14 0x4a0c1e41 in llvm::TailDuplicator::tailDuplicateAndUpdate(bool, llvm::MachineBasicBlock*, llvm::MachineBasicBlock*, llvm::SmallVectorImpl<llvm::MachineBasicBlock*>*, llvm::function_ref<void (llvm::MachineBasicBlock*)>*, llvm::SmallVectorImpl<llvm::MachineBasicBlock*>*) build_docker/../contrib/llvm/llvm/lib/CodeGen/TailDuplicator.cpp:189:5
        15 0x4a0ca16e in llvm::TailDuplicator::tailDuplicateBlocks() build_docker/../contrib/llvm/llvm/lib/CodeGen/TailDuplicator.cpp:288:19
        16 0x4a0be9f9 in (anonymous namespace)::TailDuplicateBase::runOnMachineFunction(llvm::MachineFunction&) build_docker/../contrib/llvm/llvm/lib/CodeGen/TailDuplication.cpp:98:21
        17 0x499a2777 in llvm::MachineFunctionPass::runOnFunction(llvm::Function&) build_docker/../contrib/llvm/llvm/lib/CodeGen/MachineFunctionPass.cpp:72:13
        18 0x4dbba34d in llvm::FPPassManager::runOnFunction(llvm::Function&) build_docker/../contrib/llvm/llvm/lib/IR/LegacyPassManager.cpp:1435:27
        19 0x4dbe3761 in llvm::FPPassManager::runOnModule(llvm::Module&) build_docker/../contrib/llvm/llvm/lib/IR/LegacyPassManager.cpp:1481:16
        20 0x4dbbebbb in (anonymous namespace)::MPPassManager::runOnModule(llvm::Module&) build_docker/../contrib/llvm/llvm/lib/IR/LegacyPassManager.cpp:1550:27
        21 0x4dbbebbb in llvm::legacy::PassManagerImpl::run(llvm::Module&) build_docker/../contrib/llvm/llvm/lib/IR/LegacyPassManager.cpp:541:44
        22 0x4dbe454b in llvm::legacy::PassManager::run(llvm::Module&) build_docker/../contrib/llvm/llvm/lib/IR/LegacyPassManager.cpp:1677:14
        23 0x414405df in DB::JITCompiler::compile(llvm::Module&) build_docker/../src/Interpreters/JIT/CHJIT.cpp:78:22
        24 0x4143bb7d in DB::CHJIT::compileModule(std::__1::unique_ptr<llvm::Module, std::__1::default_delete<llvm::Module>>) build_docker/../src/Interpreters/JIT/CHJIT.cpp:378:29
        25 0x4143aded in DB::CHJIT::compileModule(std::__1::function<void (llvm::Module&)>) build_docker/../src/Interpreters/JIT/CHJIT.cpp:359:24
        26 0x4147b25e in DB::compileAggregateFunctions(DB::CHJIT&, std::__1::vector<DB::AggregateFunctionWithOffset, std::__1::allocator<DB::AggregateFunctionWithOffset>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>) build_docker/../src/Interpreters/JIT/compileFunction.cpp:738:32
        27 0x3de0a23a in DB::Aggregator::compileAggregateFunctionsIfNeeded()::$_0::operator()() const build_docker/../src/Interpreters/Aggregator.cpp:661:49
        28 0x3de0a23a in std::__1::pair<std::__1::shared_ptr<DB::CompiledExpressionCacheEntry>, bool> DB::CacheBase<wide::integer<128ul, unsigned int>, DB::CompiledExpressionCacheEntry, UInt128Hash, DB::CompiledFunctionWeightFunction>::getOrSet<DB::Aggregator::compileAggregateFunctionsIfNeeded()::$_0>(wide::integer<128ul, unsigned int> const&, DB::Aggregator::compileAggregateFunctionsIfNeeded()::$_0&&) build_docker/../src/Common/CacheBase.h:125:24
        29 0x3de0a23a in DB::Aggregator::compileAggregateFunctionsIfNeeded() build_docker/../src/Interpreters/Aggregator.cpp:657:70

      Memory was marked as uninitialized
        0 0xb988ded in __sanitizer_dtor_callback (/usr/bin/clickhouse+0xb988ded) (BuildId: c4a880b742797a1f37bc4f5ed869f055cc86486b)
        1 0x498145da in llvm::MachineBasicBlock::~MachineBasicBlock() build_docker/../contrib/llvm/llvm/lib/CodeGen/MachineBasicBlock.cpp:56:1
        2 0x49969c1f in llvm::MachineFunction::DeleteMachineBasicBlock(llvm::MachineBasicBlock*) build_docker/../contrib/llvm/llvm/lib/CodeGen/MachineFunction.cpp:426:8
        3 0x49969c1f in llvm::ilist_alloc_traits<llvm::MachineBasicBlock>::deleteNode(llvm::MachineBasicBlock*) build_docker/../contrib/llvm/llvm/lib/CodeGen/MachineFunction.cpp:127:21
        4 0x4983f4d6 in llvm::iplist_impl<llvm::simple_ilist<llvm::MachineBasicBlock>, llvm::ilist_traits<llvm::MachineBasicBlock>>::erase(llvm::ilist_iterator<llvm::ilist_detail::node_options<llvm::MachineBasicBlock, false, false, void>, false, false>) build_docker/../contrib/llvm/llvm/include/llvm/ADT/ilist.h:268:11
        5 0x4983f4d6 in llvm::iplist_impl<llvm::simple_ilist<llvm::MachineBasicBlock>, llvm::ilist_traits<llvm::MachineBasicBlock>>::erase(llvm::MachineBasicBlock*) build_docker/../contrib/llvm/llvm/include/llvm/ADT/ilist.h:272:39
        6 0x4983f4d6 in llvm::MachineFunction::erase(llvm::MachineBasicBlock*) build_docker/../contrib/llvm/llvm/include/llvm/CodeGen/MachineFunction.h:767:53
        7 0x4983f4d6 in llvm::MachineBasicBlock::eraseFromParent() build_docker/../contrib/llvm/llvm/lib/CodeGen/MachineBasicBlock.cpp:1317:16
        8 0x4a0c9a27 in llvm::TailDuplicator::removeDeadBlock(llvm::MachineBasicBlock*, llvm::function_ref<void (llvm::MachineBasicBlock*)>*) build_docker/../contrib/llvm/llvm/lib/CodeGen/TailDuplicator.cpp:1051:8
        9 0x4a0c1e41 in llvm::TailDuplicator::tailDuplicateAndUpdate(bool, llvm::MachineBasicBlock*, llvm::MachineBasicBlock*, llvm::SmallVectorImpl<llvm::MachineBasicBlock*>*, llvm::function_ref<void (llvm::MachineBasicBlock*)>*, llvm::SmallVectorImpl<llvm::MachineBasicBlock*>*) build_docker/../contrib/llvm/llvm/lib/CodeGen/TailDuplicator.cpp:189:5
        10 0x4a0ca16e in llvm::TailDuplicator::tailDuplicateBlocks() build_docker/../contrib/llvm/llvm/lib/CodeGen/TailDuplicator.cpp:288:19
        11 0x4a0be9f9 in (anonymous namespace)::TailDuplicateBase::runOnMachineFunction(llvm::MachineFunction&) build_docker/../contrib/llvm/llvm/lib/CodeGen/TailDuplication.cpp:98:21
        12 0x499a2777 in llvm::MachineFunctionPass::runOnFunction(llvm::Function&) build_docker/../contrib/llvm/llvm/lib/CodeGen/MachineFunctionPass.cpp:72:13
        13 0x4dbba34d in llvm::FPPassManager::runOnFunction(llvm::Function&) build_docker/../contrib/llvm/llvm/lib/IR/LegacyPassManager.cpp:1435:27
        14 0x4dbe3761 in llvm::FPPassManager::runOnModule(llvm::Module&) build_docker/../contrib/llvm/llvm/lib/IR/LegacyPassManager.cpp:1481:16
        15 0x4dbbebbb in (anonymous namespace)::MPPassManager::runOnModule(llvm::Module&) build_docker/../contrib/llvm/llvm/lib/IR/LegacyPassManager.cpp:1550:27
        16 0x4dbbebbb in llvm::legacy::PassManagerImpl::run(llvm::Module&) build_docker/../contrib/llvm/llvm/lib/IR/LegacyPassManager.cpp:541:44
        17 0x4dbe454b in llvm::legacy::PassManager::run(llvm::Module&) build_docker/../contrib/llvm/llvm/lib/IR/LegacyPassManager.cpp:1677:14
        18 0x414405df in DB::JITCompiler::compile(llvm::Module&) build_docker/../src/Interpreters/JIT/CHJIT.cpp:78:22
        19 0x4143bb7d in DB::CHJIT::compileModule(std::__1::unique_ptr<llvm::Module, std::__1::default_delete<llvm::Module>>) build_docker/../src/Interpreters/JIT/CHJIT.cpp:378:29
        20 0x4143aded in DB::CHJIT::compileModule(std::__1::function<void (llvm::Module&)>) build_docker/../src/Interpreters/JIT/CHJIT.cpp:359:24
        21 0x4147b25e in DB::compileAggregateFunctions(DB::CHJIT&, std::__1::vector<DB::AggregateFunctionWithOffset, std::__1::allocator<DB::AggregateFunctionWithOffset>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>) build_docker/../src/Interpreters/JIT/compileFunction.cpp:738:32
        22 0x3de0a23a in DB::Aggregator::compileAggregateFunctionsIfNeeded()::$_0::operator()() const build_docker/../src/Interpreters/Aggregator.cpp:661:49
        23 0x3de0a23a in std::__1::pair<std::__1::shared_ptr<DB::CompiledExpressionCacheEntry>, bool> DB::CacheBase<wide::integer<128ul, unsigned int>, DB::CompiledExpressionCacheEntry, UInt128Hash, DB::CompiledFunctionWeightFunction>::getOrSet<DB::Aggregator::compileAggregateFunctionsIfNeeded()::$_0>(wide::integer<128ul, unsigned int> const&, DB::Aggregator::compileAggregateFunctionsIfNeeded()::$_0&&) build_docker/../src/Common/CacheBase.h:125:24
        24 0x3de0a23a in DB::Aggregator::compileAggregateFunctionsIfNeeded() build_docker/../src/Interpreters/Aggregator.cpp:657:70

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/41046/490a2c75610c4bc3191d55226f8454b3c3d3919a/stateful_tests__msan_.html

Note, that it is safe to do, but only for this method, since it had been
disabled anyway, back in #27574, and I guess this MSan report may be
related.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-09-10 21:38:35 +02:00
Alexey Milovidov
fd235919aa Remove some methods 2022-09-10 05:04:40 +02:00
Alexey Milovidov
fa62c7e982 Fix half of trash 2022-09-10 04:08:16 +02:00
Igor Nikonov
726639484a
Revert "Query plan optimization setting: read in window order" 2022-09-09 22:15:38 +02:00
Denny Crane
be90cecbd2 Revert "Revert "Fix trivial count optimization with array join""
This reverts commit 17de7b2876.
2022-09-09 14:56:38 -03:00
Azat Khuzhin
763bb18f98 Fix SIGSEGV in SortedBlocksWriter in case of empty block
CI found one issue [1].

Here is the stack trace for invalid read:

<details>

<summary>stack trace</summary>

```
    0: DB::TemporaryFileLazySource::TemporaryFileLazySource(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Block const&) [inlined] std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::__is_long(this="") const at string:1445:22
    1: DB::TemporaryFileLazySource::TemporaryFileLazySource(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Block const&) [inlined] std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::basic_string(this="", __str="") at string:1927
    2: DB::TemporaryFileLazySource::TemporaryFileLazySource(this=0x00007f3aec105f58, path_="", header_=0x00007f38ffd93b40) at TemporaryFileLazySource.cpp:11
    3: DB::SortedBlocksWriter::streamFromFile(std::__1::unique_ptr<Poco::TemporaryFile, std::__1::default_delete<Poco::TemporaryFile> > const&) const [inlined] DB::TemporaryFileLazySource* std::__1::construct_at<DB::TemporaryFileLazySource, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Block, DB::TemporaryFileLazySource*>(__args=0x00007f38ffd91560) at construct_at.h:38:50
    4: DB::SortedBlocksWriter::streamFromFile(std::__1::unique_ptr<Poco::TemporaryFile, std::__1::default_delete<Poco::TemporaryFile> > const&) const [inlined] void std::__1::allocator_traits<std::__1::allocator<DB::TemporaryFileLazySource> >::construct<DB::TemporaryFileLazySource, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Block, void, void>(__args=0x00007f38ffd91560) at allocator_traits.h:298
    5: DB::SortedBlocksWriter::streamFromFile(std::__1::unique_ptr<Poco::TemporaryFile, std::__1::default_delete<Poco::TemporaryFile> > const&) const [inlined] std::__1::__shared_ptr_emplace<DB::TemporaryFileLazySource, std::__1::allocator<DB::TemporaryFileLazySource> >::__shared_ptr_emplace<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Block>(this=0x00007f3aec105f40, __args=0x00007f38ffd91560) at shared_ptr.h:293
    6: DB::SortedBlocksWriter::streamFromFile(std::__1::unique_ptr<Poco::TemporaryFile, std::__1::default_delete<Poco::TemporaryFile> > const&) const [inlined] std::__1::shared_ptr<DB::TemporaryFileLazySource> std::__1::allocate_shared<DB::TemporaryFileLazySource, std::__1::allocator<DB::TemporaryFileLazySource>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Block, void>(__args=<unavailable>, __args=<unavailable>) at shared_ptr.h:954
    7: DB::SortedBlocksWriter::streamFromFile(std::__1::unique_ptr<Poco::TemporaryFile, std::__1::default_delete<Poco::TemporaryFile> > const&) const [inlined] std::__1::shared_ptr<DB::TemporaryFileLazySource> std::__1::make_shared<DB::TemporaryFileLazySource, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Block, void>(__args=<unavailable>, __args=<unavailable>) at shared_ptr.h:963
    8: DB::SortedBlocksWriter::streamFromFile(this=<unavailable>, file=<unavailable>) const at SortedBlocksWriter.cpp:238
    9: DB::SortedBlocksWriter::premerge(this=<unavailable>) at SortedBlocksWriter.cpp:209:32
```

</details>

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/41046/adea92f847373d1fcfd733d8979c63024f9b80bf/stress_test__asan_.html

So the problem here is that there was empty unique_ptr<> reference to
temporary file, because of empty block that accepted by
SortedBlocksWriter::insert(), but insert() is not a problem the problem
is premerge() that steals blocks from insert() and do not have check
that there are some rows. However this check exists in
SortedBlocksWriter::flush(), and in that case temporary file is not
created.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-09-09 19:52:38 +02:00
Azat Khuzhin
57146c9361 Fix typos in SortedBlocksWriter
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-09-09 19:52:38 +02:00
Anton Popov
a1b5582a7c
Merge pull request #37152 from CurtizJ/fix-filling-of-missed-nested
Fix filling of missed Nested columns with multiple levels
2022-09-09 19:29:01 +02:00
Alexander Tokmakov
9bcec2d49a stop background ops before converting db engine 2022-09-09 18:27:19 +02:00
zhenjial
51badf3d0a fix 2022-09-10 00:08:50 +08:00
Dmitry Novik
64c4023737
Merge branch 'master' into grouping-comp 2022-09-09 18:05:01 +02:00
zhenjial
bd9fabc3f7 code optimization, add test 2022-09-09 23:27:42 +08:00
avogar
ad68b7be0f Better 2022-09-09 15:01:45 +00:00
Kseniia Sumarokova
5bb40e5856
Change cache setting do_not_evict_index_and_mark_files default to 0. (#41139) 2022-09-09 15:04:32 +02:00
Antonio Andelic
32483aeec8 Merge branch 'master' into keeper-storage 2022-09-09 12:24:01 +00:00
kssenii
5cba9b789c Merge remote-tracking branch 'upstream/master' into refactor-cache-for-async-download 2022-09-09 14:14:04 +02:00
Igor Nikonov
5379f37331
Merge pull request #41015 from ClickHouse/window_func_optimization_setting
Query plan optimization setting: read in window order
2022-09-09 14:00:41 +02:00
Raúl Marín
b33c1fa60f Fix query_views_log with Window views 2022-09-09 13:46:53 +02:00
kssenii
6a885430d7 Add assertion 2022-09-09 13:36:23 +02:00
kssenii
c2c35278ba Refactor cache for async download 2022-09-09 13:12:47 +02:00
lthaooo
d4dd6b05b2 optimize compact part mutation 2022-09-09 10:22:35 +08:00
zhenjial
469ceaa156 code optimization 2022-09-09 00:47:43 +08:00
Kruglov Pavel
93d38b9451
Fix style 2022-09-08 13:19:18 +02:00
Robert Schulze
c16707ff00
chore: delete obsolete modelEvaluate() function + SYSTEM.MODELS view
- The deleted function modelEvaluate() was superseded by
  catboostEvaluate().

- Also delete the external model repository, as modelEvaluate() was it's
  last user. Additionally remove the system view SYSTEM.MODELS for
  inspecting the repository.

- SYSTEM RELOAD MODELS is also obsolete. HOWEVER, it was retained and
  made a no-op instead of deleted.

  Why?
  The reason is that RBAC in distributed setups works by storing
  privileges (granted and revoked) as plain SQL statements in Keeper.
  Nodes read these statements at startup and parse them. If a privilege
  for SYSTEM RELOAD MODELS exists but parser doesn't recognize it
  nodes would fail to come up.

  Considered but rejected alternatives:
  - Ignore SYSTEM RELOAD MODELS during parsing RBAC privileges and
    return an error for regular SYSTEM RELOAD MODELS SQL. Special-case
    of no-op behavior, too brittle.
  - Remove SYSTEM RELOAD MODELS manually from Keeper via command-line
    manipulation of Keeper nodes or via SQL by dropping the privileges.
    Needs user intervention during upgrade.
2022-09-08 09:10:11 +00:00
Anton Popov
7c12b448b8 Merge remote-tracking branch 'upstream/master' into HEAD 2022-09-08 01:52:52 +00:00
Anton Popov
5a790b15b4 try to fix filling of missed Nested columns with multiple levels 2022-09-08 00:20:11 +00:00
Igor Nikonov
e74bb00e90 Fix: EXPLAIN PLAN - make settings in SETTING clause effective 2022-09-07 19:11:25 +00:00
avogar
0475eb0b17 Make better 2022-09-07 18:53:22 +00:00
avogar
5c0c19c533 Make generic solution 2022-09-07 18:49:34 +00:00
kssenii
e51313b6b3 Get rid of static threadpools 2022-09-07 17:48:11 +02:00
avogar
9da22f7b7c Fix possible use-heap-after-free in -Map combinator 2022-09-07 15:06:56 +00:00
Antonio Andelic
d15fdae8dc
Merge pull request #40853 from ClickHouse/embeddedrocksdb-delete-update-support
Support for `UPDATE` and `DELETE` in `EmbeddedRocksDB`
2022-09-07 11:03:49 +02:00
kssenii
487bc0fba3 Fix heap use after free 2022-09-06 16:50:58 +02:00
Anton Popov
f76c1482bd try to fix filling of missed Nested columns with multiple levels 2022-09-06 13:56:32 +00:00
Filatenkov Artur
7949b3fb74
Merge branch 'master' into annoy 2022-09-06 11:23:15 +03:00
Yong Wang
d5337fd813
Merge branch 'master' into Kusto-phase1 2022-09-06 00:51:39 -04:00
Alexey Milovidov
940a53e519
Merge pull request #40984 from Lucky-Chang/typo_fix
Fix some typos and clang-tidy warnings
2022-09-06 02:37:50 +03:00
Yong Wang
2ca54d12ee
Merge branch 'ClickHouse:master' into Kusto-phase1 2022-09-05 17:42:38 -04:00
kssenii
4c8aa04b73 Revert "Fix"
This reverts commit ec55c90f36.
2022-09-05 20:12:40 +02:00
ianton-ru
39e1fc7a0f
Merge branch 'master' into fix-ordinary-system-unfreeze 2022-09-05 17:10:59 +03:00
FArthur-cmd
26eb6cdaec refactoring 2022-09-05 13:04:42 +00:00
FArthur-cmd
9096b73084 add setting 2022-09-05 12:41:46 +00:00
Nikita Mikhaylov
33e530392e
Check grants for databases in case of renaming databases (#40906) 2022-09-05 12:38:44 +02:00
Antonio Andelic
e07539d1b9 Merge branch 'master' into keeper-storage 2022-09-05 09:54:13 +00:00
Luck-Chang
1ac8e739c9 fix some typos and clang-tidy warnings 2022-09-05 09:50:24 +08:00
Nikita Taranov
16af4aebc8
impl (#40952) 2022-09-04 16:26:31 +02:00
Alexey Milovidov
193cd1b3b2
Merge pull request #39138 from nickitat/control_block_size_in_aggregator
Control block size in aggregator
2022-09-04 04:51:00 +03:00
Yakov Olkhovskiy
fdcced8962
Merge pull request #40850 from canhld94/ch_canh_fix_set
Fix tryGetPreparedSet crash when same set expression built from different column(s)
2022-09-02 14:40:35 -04:00
kssenii
2dcf3fef94 Fix tests 2022-09-02 19:27:52 +02:00
Kseniia Sumarokova
7dea71c83f
Merge pull request #40884 from TKaxe/fix_removeUnneededColumn
fix bug when remove unneeded columns in subquery
2022-09-02 18:51:18 +02:00
Kseniia Sumarokova
1370bf0de7
Merge pull request #40820 from kssenii/cache-assert-directory-ownership
Create status file for cache directory
2022-09-02 18:42:11 +02:00
Anton Popov
9dda9658a8 Merge remote-tracking branch 'upstream/master' into HEAD 2022-09-02 12:48:27 +00:00
Vladimir C
963c0111bf
Merge pull request #39418 from vdimir/join_and_sets
Filter joined streams for `full_sorting_join` by each other before sorting
2022-09-02 13:57:06 +02:00
Antonio Andelic
9ca20b97c2
Merge branch 'master' into keeper-storage 2022-09-02 12:13:07 +02:00
Antonio Andelic
e64436fef3 Fix typos with new codespell 2022-09-02 08:54:48 +00:00
Robert Schulze
319d8b00a7
Merge pull request #39010 from FrankChen021/tracing_context_propagation
Improve the opentelemetry tracing context propagation across threads
2022-09-02 07:56:43 +02:00
Duc Canh Le
65b64010a0 move out-of-bound check into set::areTypesEqual 2022-09-02 11:56:13 +08:00
Dmitry Novik
ddadb362cf
Merge pull request #39762 from quickhouse/betterorderbyoptimization
Fixed `Unknown identifier (aggregate-function)` exception which appears when a user tries to calculate WINDOW ORDER BY/PARTITION BY expressions over aggregate functions
2022-09-01 18:08:06 +02:00
Frank Chen
9d63cbe811 Merge 'origin/master' into tracing_context_propagation to resolve conflicts 2022-09-01 23:18:59 +08:00
Frank Chen
1da35a486c Disable query_span for internal queries 2022-09-01 23:16:05 +08:00
kssenii
ec55c90f36 Fix 2022-09-01 17:04:34 +02:00
Antonio Andelic
9231084c57 Style fix 2022-09-01 14:40:50 +00:00
Antonio Andelic
a0c1ce5eb7 Merge branch 'master' into embeddedrocksdb-delete-update-support 2022-09-01 14:40:24 +00:00
Vladimir C
12e6fc4182
Merge branch 'master' into join_and_sets 2022-09-01 14:56:14 +02:00
Vladimir C
f2cf7d7762
Merge pull request #40334 from ClickHouse/vdimir/analyze-stuck-limit
Limit number of analyze for one query, att. 2
2022-09-01 14:40:20 +02:00
Kseniia Sumarokova
c6c67a248d
Merge pull request #40792 from canhld94/ch_canh_intersect_distinct
Implement intersect + except distinct
2022-09-01 14:35:26 +02:00
Antonio Andelic
62ff61404f Improve remove recursive 2022-09-01 12:16:13 +00:00
kssenii
82879543d3 Merge remote-tracking branch 'upstream/master' into improve-marks-cache-loading 2022-09-01 13:13:01 +02:00
kssenii
9ae95f8c04 Fix unit test 2022-09-01 13:11:47 +02:00
kssenii
ca4363e2e3 Merge master 2022-09-01 13:09:00 +02:00
Anton Popov
f7bdf07adc
Merge pull request #38715 from CurtizJ/fix-read-in-order-fixed-prefix
Better support of `optimize_read_in_order` in case of fixed prefix of sorting key
2022-09-01 12:59:18 +02:00
kssenii
0d1f7afb17 Merge remote-tracking branch 'upstream/master' into fix-removing-directories 2022-09-01 12:28:51 +02:00