Commit Graph

97506 Commits

Author SHA1 Message Date
Rich Raposa
34c2c4bb52
Merge branch 'master' into lightweight-delete 2022-09-09 12:13:51 -06:00
Nikolay Degterinsky
6dcaa9f14d
Merge pull request #41106 from vincentbernat/fix/docs-sync
docs: mention SYNC modifier for DROP and DETACH statements
2022-09-09 20:12:47 +02:00
Azat Khuzhin
8c5583d7a5 Fix stack-use-after-return in GetPriorityForLoadBalancing::getPriorityFunc()
clang-15 reports [1]:

<details>

<summary>ASan report</summary>

```
    ==1==ERROR: AddressSanitizer: stack-use-after-return on address 0x7f1d04c4eb20 at pc 0x000031c4803c bp 0x7f1d05e19a00 sp 0x7f1d05e199f8
    READ of size 8 at 0x7f1d04c4eb20 thread T200 (QueryPullPipeEx)
        #0 0x31c4803b in DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3::operator()(unsigned long) const build_docker/../src/Common/GetPriorityForLoadBalancing.cpp:42:40
        #1 0x31c4803b in decltype(static_cast<DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3&>(fp)(static_cast<unsigned long>(fp0))) std::__1::__invoke<DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3&, unsigned long>(DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3&, unsigned long&&) build_docker/../contrib/libcxx/include/type_traits:3640:23
        #2 0x31c4803b in unsigned long std::__1::__invoke_void_return_wrapper<unsigned long, false>::__call<DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3&, unsigned long>(DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3&, unsigned long&&) build_docker/../contrib/libcxx/include/__functional/invoke.h:30:16
        #3 0x31c4803b in std::__1::__function::__default_alloc_func<DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3, unsigned long (unsigned long)>::operator()(unsigned long&&) build_docker/../contrib/libcxx/include/__functional/function.h:230:12
        #4 0x31c4803b in unsigned long std::__1::__function::__policy_invoker<unsigned long (unsigned long)>::__call_impl<std::__1::__function::__default_alloc_func<DB::GetPriorityForLoadBalancing::getPriorityFunc(DB::LoadBalancing, unsigned long, unsigned long) const::$_3, unsigned long (unsigned long)>>(std::__1::__function::__policy_storage const*, unsigned long) build_docker/../contrib/libcxx/include/__functional/function.h:711:16
        #5 0x31c38b07 in std::__1::__function::__policy_func<unsigned long (unsigned long)>::operator()(unsigned long&&) const build_docker/../contrib/libcxx/include/__functional/function.h:843:16
        #6 0x31c38b07 in std::__1::function<unsigned long (unsigned long)>::operator()(unsigned long) const build_docker/../contrib/libcxx/include/__functional/function.h:1184:12
        #7 0x31c38b07 in PoolWithFailoverBase<DB::IConnectionPool>::getShuffledPools(unsigned long, std::__1::function<unsigned long (unsigned long)> const&) build_docker/../src/Common/PoolWithFailoverBase.h:174:39

      This frame has 2 object(s):
        [32, 40) 'pool_size.addr' <== Memory access at offset 32 is inside this variable
        [64, 88) 'ref.tmp' (line 18)
```

</details>

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/41046/adea92f847373d1fcfd733d8979c63024f9b80bf/integration_tests__asan__[1/3].html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-09-09 20:06:48 +02:00
Denny Crane
ac96f29ae1 make the test to trick fuzzer 2022-09-09 15:00:14 -03:00
Denny Crane
be90cecbd2 Revert "Revert "Fix trivial count optimization with array join""
This reverts commit 17de7b2876.
2022-09-09 14:56:38 -03:00
Azat Khuzhin
62a105f216 Update capnproto to v0.10.2 (to address issue with ASan in clang-15)
This should address issue with ASan:

- CI report - https://s3.amazonaws.com/clickhouse-test-reports/41046/490a2c75610c4bc3191d55226f8454b3c3d3919a/stateless_tests__asan__[1/2].html

    2022-09-08 19:39:40 kj/exception.c++:977: failed: expected offset < 65536 && offset > -65536; ExceptionCallback must be allocated on the stack.

- Discussion in ML - https://www.mail-archive.com/capnproto@googlegroups.com/msg01451.html

- Fix capnproto/capnproto@c4eef80a13

    "Fix ASAN problems under Clang 15.

    It appears ASAN now by default tries to detect stack-use-after-return. This breaks our assumptions in requireOnStack() and totally breaks fibers.

    For requireOnStack() we can just skip the check in this case.

    For fibers, we need to implement the ASAN hints to tell it when we're switching fibers."

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-09-09 19:56:25 +02:00
Azat Khuzhin
763bb18f98 Fix SIGSEGV in SortedBlocksWriter in case of empty block
CI found one issue [1].

Here is the stack trace for invalid read:

<details>

<summary>stack trace</summary>

```
    0: DB::TemporaryFileLazySource::TemporaryFileLazySource(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Block const&) [inlined] std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::__is_long(this="") const at string:1445:22
    1: DB::TemporaryFileLazySource::TemporaryFileLazySource(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Block const&) [inlined] std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::basic_string(this="", __str="") at string:1927
    2: DB::TemporaryFileLazySource::TemporaryFileLazySource(this=0x00007f3aec105f58, path_="", header_=0x00007f38ffd93b40) at TemporaryFileLazySource.cpp:11
    3: DB::SortedBlocksWriter::streamFromFile(std::__1::unique_ptr<Poco::TemporaryFile, std::__1::default_delete<Poco::TemporaryFile> > const&) const [inlined] DB::TemporaryFileLazySource* std::__1::construct_at<DB::TemporaryFileLazySource, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Block, DB::TemporaryFileLazySource*>(__args=0x00007f38ffd91560) at construct_at.h:38:50
    4: DB::SortedBlocksWriter::streamFromFile(std::__1::unique_ptr<Poco::TemporaryFile, std::__1::default_delete<Poco::TemporaryFile> > const&) const [inlined] void std::__1::allocator_traits<std::__1::allocator<DB::TemporaryFileLazySource> >::construct<DB::TemporaryFileLazySource, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Block, void, void>(__args=0x00007f38ffd91560) at allocator_traits.h:298
    5: DB::SortedBlocksWriter::streamFromFile(std::__1::unique_ptr<Poco::TemporaryFile, std::__1::default_delete<Poco::TemporaryFile> > const&) const [inlined] std::__1::__shared_ptr_emplace<DB::TemporaryFileLazySource, std::__1::allocator<DB::TemporaryFileLazySource> >::__shared_ptr_emplace<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Block>(this=0x00007f3aec105f40, __args=0x00007f38ffd91560) at shared_ptr.h:293
    6: DB::SortedBlocksWriter::streamFromFile(std::__1::unique_ptr<Poco::TemporaryFile, std::__1::default_delete<Poco::TemporaryFile> > const&) const [inlined] std::__1::shared_ptr<DB::TemporaryFileLazySource> std::__1::allocate_shared<DB::TemporaryFileLazySource, std::__1::allocator<DB::TemporaryFileLazySource>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Block, void>(__args=<unavailable>, __args=<unavailable>) at shared_ptr.h:954
    7: DB::SortedBlocksWriter::streamFromFile(std::__1::unique_ptr<Poco::TemporaryFile, std::__1::default_delete<Poco::TemporaryFile> > const&) const [inlined] std::__1::shared_ptr<DB::TemporaryFileLazySource> std::__1::make_shared<DB::TemporaryFileLazySource, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Block, void>(__args=<unavailable>, __args=<unavailable>) at shared_ptr.h:963
    8: DB::SortedBlocksWriter::streamFromFile(this=<unavailable>, file=<unavailable>) const at SortedBlocksWriter.cpp:238
    9: DB::SortedBlocksWriter::premerge(this=<unavailable>) at SortedBlocksWriter.cpp:209:32
```

</details>

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/41046/adea92f847373d1fcfd733d8979c63024f9b80bf/stress_test__asan_.html

So the problem here is that there was empty unique_ptr<> reference to
temporary file, because of empty block that accepted by
SortedBlocksWriter::insert(), but insert() is not a problem the problem
is premerge() that steals blocks from insert() and do not have check
that there are some rows. However this check exists in
SortedBlocksWriter::flush(), and in that case temporary file is not
created.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-09-09 19:52:38 +02:00
Azat Khuzhin
57146c9361 Fix typos in SortedBlocksWriter
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-09-09 19:52:38 +02:00
Anton Popov
a1b5582a7c
Merge pull request #37152 from CurtizJ/fix-filling-of-missed-nested
Fix filling of missed Nested columns with multiple levels
2022-09-09 19:29:01 +02:00
Alexander Tokmakov
e77b9e4d0c
Merge pull request #40775 from azat/ci/core-dumps-rework
Rework core collecting on CI (eliminate gcore usage)
2022-09-09 20:20:10 +03:00
Igor Nikonov
4491b4f29b
Merge branch 'master' into fix_int8_test_standalone_keeper_mode 2022-09-09 19:11:14 +02:00
zhenjial
da8e5631ea fix test 2022-09-10 00:47:34 +08:00
Alexander Tokmakov
9bcec2d49a stop background ops before converting db engine 2022-09-09 18:27:19 +02:00
zhenjial
51badf3d0a fix 2022-09-10 00:08:50 +08:00
Dmitry Novik
64c4023737
Merge branch 'master' into grouping-comp 2022-09-09 18:05:01 +02:00
Dmitry Novik
877776e569
Update src/Functions/grouping.h 2022-09-09 18:04:29 +02:00
Kruglov Pavel
c33aa54032
Fix 2022-09-09 17:53:26 +02:00
zhenjial
bd9fabc3f7 code optimization, add test 2022-09-09 23:27:42 +08:00
Mikhail f. Shiryaev
b815fdbc8c
Merge pull request #41088 from ClickHouse/darwin-releases
Add macos binaries to release assets
2022-09-09 17:17:15 +02:00
Nikolay Degterinsky
12e7cd6b19
Merge pull request #41068 from jthmath/bugfix
fix ignore projection logic error
2022-09-09 17:10:03 +02:00
alesapin
86a3719bef
Merge pull request #41128 from ClickHouse/less_errors
Fix bad warnings in part fetches
2022-09-09 16:58:09 +02:00
alesapin
e904822051 Bump minio version in tests 2022-09-09 16:54:03 +02:00
Igor Nikonov
dbfb448290 Fix: integration test, standalone keeper mode
There is logic regarding which keeper binary use to start keeper cluster in an integration test
There 2 options:
(1) standalone keeper binary (expected binary name clickhouse-keeper)
(2) clickhouse binary with keeper inside

Fixed:
- option (1) didn't work since docker_compose_keeper.yaml didn't create
target clickhouse-keeper at all
- if clickhouse-keeper existed, option (1) was taken but
  clickhouse-keeper could be just a link to clickhouse binary (the link
  is created always during build if cmake option BUILD_STANDALONE_KEEPER is OFF)
2022-09-09 14:51:34 +00:00
Alexander Tokmakov
98509e62e8
Update src/Storages/Freeze.cpp 2022-09-09 16:20:24 +03:00
Kseniia Sumarokova
5bb40e5856
Change cache setting do_not_evict_index_and_mark_files default to 0. (#41139) 2022-09-09 15:04:32 +02:00
Dan Roscigno
2b36de3818
Merge pull request #41138 from DanRoscigno/move-install-title-to-frontmatter
move title to frontmatter to allow inclusion in other docs
2022-09-09 09:03:11 -04:00
DanRoscigno
afe3717761 move title to frontmatter to allow inclusion in other docs 2022-09-09 08:34:42 -04:00
Alexander Tokmakov
3df65b9cfb
Update 01172_transaction_counters.sql 2022-09-09 15:25:52 +03:00
Antonio Andelic
2472b9c404 Mark primary key test as long 2022-09-09 12:24:26 +00:00
Antonio Andelic
32483aeec8 Merge branch 'master' into keeper-storage 2022-09-09 12:24:01 +00:00
kssenii
5cba9b789c Merge remote-tracking branch 'upstream/master' into refactor-cache-for-async-download 2022-09-09 14:14:04 +02:00
Igor Nikonov
5379f37331
Merge pull request #41015 from ClickHouse/window_func_optimization_setting
Query plan optimization setting: read in window order
2022-09-09 14:00:41 +02:00
Raúl Marín
b33c1fa60f Fix query_views_log with Window views 2022-09-09 13:46:53 +02:00
alesapin
01e745bbbf Fix parts removal 2022-09-09 13:26:13 +02:00
Alexander Tokmakov
d6dbde4289
Merge pull request #41066 from vitlibar/no-hardlinks-while-making-backup-of-mergetree-in-atomic-db
No hardlinks while making backup of MergeTree in atomic database.
2022-09-09 14:25:11 +03:00
Alexander Tokmakov
48927ba0ac
Merge branch 'master' into no-hardlinks-while-making-backup-of-mergetree-in-atomic-db 2022-09-09 14:24:44 +03:00
alesapin
23d95e6723 Fix parts removal in case of failures 2022-09-09 13:24:22 +02:00
kssenii
c2c35278ba Refactor cache for async download 2022-09-09 13:12:47 +02:00
Frank Chen
2fb0ae7002 Update test case 2022-09-09 19:02:42 +08:00
alesapin
a713c5ac27
Merge pull request #41126 from azat/tests/01650_fetch_patition_with_macro_in_zk_path_lon
tests: disable zero copy replication to suppress warning in 01650_fetch_patition_with_macro_in_zk_path_long
2022-09-09 12:23:29 +02:00
alesapin
e3fc78aa5f Fix bad warnings in part fetches 2022-09-09 12:14:42 +02:00
Azat Khuzhin
927f763cbb tests: disable zero copy replication to suppress warning in 01650_fetch_patition_with_macro_in_zk_path_long
Fixes: #41108
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-09-09 10:12:32 +02:00
Antonio Andelic
8ae7fb14b3
Merge pull request #40918 from ClickHouse/fix-keeper-system-path-check
Fix check for Keeper system path modification
2022-09-09 09:12:49 +02:00
Antonio Andelic
43b6f395fb Mark the test as long 2022-09-09 06:38:44 +00:00
Frank Chen
92a92baa33 Simplify test
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
2022-09-09 12:18:27 +08:00
Frank Chen
237abffdba Improve test
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
2022-09-09 12:02:02 +08:00
lthaooo
d4dd6b05b2 optimize compact part mutation 2022-09-09 10:22:35 +08:00
rfraposa
18f5b5e5b0 Add docs for lightweight deletes 2022-09-08 16:05:56 -06:00
Alexey Milovidov
af8ed9c166
Merge branch 'master' into sensitive_data_masker_unittest_issue 2022-09-09 00:05:53 +03:00
Alexey Milovidov
9e8e66c567
Merge pull request #41025 from den-crane/test/column_name_primary
test for column name primary
2022-09-09 00:04:53 +03:00