Commit Graph

74067 Commits

Author SHA1 Message Date
Kseniia Sumarokova
3663d23e4d
Merge pull request #29731 from kssenii/exists-subquery
Support EXISTS(subquery)
2021-10-05 22:26:19 +03:00
Azat Khuzhin
ae5570e6d6 Improve parallel_view_processing test 2021-10-05 22:20:46 +03:00
Azat Khuzhin
99ebdbb16e Fix parallel_view_processing
Number of threads got lost during after QueryPipelineBuilder::addChains()
2021-10-05 22:20:46 +03:00
romanzhukov
af94e30a95 Update H3 functions
Update en and add ru draft.
2021-10-05 22:12:44 +03:00
romanzhukov
fbfdd605ee Update metrica.md
Update ru with for hits_100m_obfuscated
2021-10-05 21:58:49 +03:00
Anton Popov
199ac31b48
Merge pull request #29683 from myrrc/improvement/tl-to-common
Non-recursive implementation for type list and its functions
2021-10-05 21:49:15 +03:00
Azat Khuzhin
4f5118b85f Fix data-race between fatal error handler and progress packets
This is not that important, but because of packets overlaps the client
may not render the fatal error.

You will find TSan report in `details`.

<details>

```
WARNING: ThreadSanitizer: data race (pid=13384)
  Write of size 8 at 0x7b24000ad818 by thread T47:
    0 DB::WriteBuffer::next() obj-x86_64-linux-gnu/../src/IO/WriteBuffer.h:43:15 (clickhouse-tsan+0x16124d61)
    1 DB::TCPHandler::sendProgress() obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:1602:10 (clickhouse-tsan+0x16124d61)
    2 DB::TCPHandler::runImpl()::$_6::operator()() const obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:318:25 (clickhouse-tsan+0x1612cf54)

  Previous write of size 8 at 0x7b24000ad818 by thread T229:
    0 DB::WriteBuffer::next() obj-x86_64-linux-gnu/../src/IO/WriteBuffer.h:43:15 (clickhouse-tsan+0x1612b793)
    1 DB::TCPHandler::sendLogData(DB::Block const&) obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:1561:10 (clickhouse-tsan+0x1612b793)
    2 DB::TCPHandler::sendLogs() obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:1632:9 (clickhouse-tsan+0x161228ab)
    3 DB::TCPHandler::runImpl()::$_1::operator()() const obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:229:62 (clickhouse-tsan+0x1612c4e1)
```

</details>
2021-10-05 21:21:32 +03:00
Azat Khuzhin
342ec02664 Fix concurrent access to LowCardinality during GROUP BY (leads to SIGSEGV)
The problem is that GROUP BY can update saved_hash, which can be also
updated by subsequent update of a dictionary, and this will lead to
use-after-free.

You will find ASan report in `details`.

<details>

    ==24679==ERROR: AddressSanitizer: heap-use-after-free on address 0x604000615d20 at pc 0x000022cc8684 bp 0x7ffea6b5f850 sp 0x7ffea6b5f848
    READ of size 8 at 0x604000615d20 thread T223 (QueryPipelineEx)
        0 0x22cc8683 in DB::ReverseIndex<>::insert(StringRef const&) obj-x86_64-linux-gnu/../src/Columns/ReverseIndex.h
        1 0x22cc0de1 in COW<DB::IColumn>::mutable_ptr<DB::IColumn> DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeImpl<char8_t>()::'lambda'()::operator()() const obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:540:39
        2 0x22cc0de1 in COW<DB::IColumn>::mutable_ptr<DB::IColumn> DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeImpl<char8_t>() obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:570:23
        3 0x22cb9c66 in COW<DB::IColumn>::mutable_ptr<DB::IColumn> DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeFrom()::'lambda'(auto)::operator()<char8_t>(auto) const obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:592:26
        4 0x22cb9c66 in DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeFrom() obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:600:28
        5 0x2500b897 in DB::ColumnLowCardinality::insertRangeFrom() obj-x86_64-linux-gnu/../src/Columns/ColumnLowCardinality.cpp:205:62
        6 0x25a182f4 in DB::appendBlock(DB::Block const&, DB::Block&) obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:470:23
        7 0x25a182f4 in DB::BufferSink::insertIntoBuffer(DB::Block const&, DB::StorageBuffer::Buffer&) obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:634:9
        8 0x25a173cc in DB::BufferSink::consume(DB::Chunk) obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:595:9
        9 0x26d1c997 in DB::SinkToStorage::transform(DB::Chunk&) obj-x86_64-linux-gnu/../src/Processors/Sinks/SinkToStorage.cpp:18:5

    0x604000615d20 is located 16 bytes inside of 40-byte region [0x604000615d10,0x604000615d38)
    freed by thread T37 (QueryPipelineEx) here:
        2 0x22cb9392 in boost::intrusive_ptr<DB::ColumnVector<unsigned long> >::~intrusive_ptr() obj-x86_64-linux-gnu/../contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:98:23
        4 0x22cb9392 in COW<DB::IColumn>::mutable_ptr<DB::ColumnVector<unsigned long> >::operator=() obj-x86_64-linux-gnu/../src/Common/COW.h💯57
        5 0x22cb9392 in DB::ReverseIndex<>::tryGetSavedHash() const obj-x86_64-linux-gnu/../src/Columns/ReverseIndex.h:362:28
        6 0x22cb9392 in DB::ColumnUnique<DB::ColumnString>::tryGetSavedHash() const obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:125:76
        7 0x242eaed3 in DB::ColumnsHashing::HashMethodSingleLowCardinalityColumn<>::HashMethodSingleLowCardinalityColumn() obj-x86_64-linux-gnu/../src/Common/ColumnsHashing.h:287:50
        8 0x242206c6 in void DB::Aggregator::executeImpl<>() const obj-x86_64-linux-gnu/../src/Interpreters/Aggregator.cpp:596:28
        9 0x24148e99 in DB::Aggregator::executeOnBlock() const obj-x86_64-linux-gnu/../src/Interpreters/Aggregator.cpp:1004:9
        10 0x26c24f3a in DB::AggregatingTransform::consume(DB::Chunk) obj-x86_64-linux-gnu/../src/Processors/Transforms/AggregatingTransform.cpp:539:33
        11 0x26c2054e in DB::AggregatingTransform::work() obj-x86_64-linux-gnu/../src/Processors/Transforms/AggregatingTransform.cpp:500:9

    previously allocated by thread T37 (QueryPipelineEx) here:
        0 0xb6d44fd in operator new(unsigned long) (/src/ch/tmp/upstream/clickhouse-asan+0xb6d44fd)
        1 0x11b78580 in COW<DB::IColumn>::mutable_ptr<> COWHelper<>::create<unsigned long&>(unsigned long&) (/src/ch/tmp/upstream/clickhouse-asan+0x11b78580)
        2 0x22cbf7b1 in DB::ReverseIndex<>::calcHashes() const obj-x86_64-linux-gnu/../src/Columns/ReverseIndex.h:472:17
        3 0x22cc2307 in DB::ReverseIndex<>::buildIndex() obj-x86_64-linux-gnu/../src/Columns/ReverseIndex.h:438:22
        4 0x22cc658c in DB::ReverseIndex<>::insert(StringRef const&) obj-x86_64-linux-gnu/../src/Columns/ReverseIndex.h:484:9
        5 0x22cc0de1 in COW<DB::IColumn>::mutable_ptr<DB::IColumn> DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeImpl<char8_t>()::'lambda'()::operator()() const obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:540:39
        6 0x22cc0de1 in COW<DB::IColumn>::mutable_ptr<DB::IColumn> DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeImpl<char8_t>() obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:570:23
        7 0x22cb9c66 in COW<DB::IColumn>::mutable_ptr<DB::IColumn> DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeFrom()::'lambda'(auto)::operator()<char8_t>(auto) const obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:592:26
        8 0x22cb9c66 in DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeFrom() obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:600:28
        9 0x2500b897 in DB::ColumnLowCardinality::insertRangeFrom() obj-x86_64-linux-gnu/../src/Columns/ColumnLowCardinality.cpp:205:62
        10 0x25a182f4 in DB::appendBlock(DB::Block const&, DB::Block&) obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:470:23
        11 0x25a182f4 in DB::BufferSink::insertIntoBuffer(DB::Block const&, DB::StorageBuffer::Buffer&) obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:634:9
        12 0x25a173cc in DB::BufferSink::consume(DB::Chunk) obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:595:9
        13 0x26d1c997 in DB::SinkToStorage::transform(DB::Chunk&) obj-x86_64-linux-gnu/../src/Processors/Sinks/SinkToStorage.cpp:18:5

    SUMMARY: AddressSanitizer: heap-use-after-free obj-x86_64-linux-gnu/../src/Columns/ReverseIndex.h in DB::ReverseIndex<unsigned long, DB::ColumnString>::insert(StringRef const&)
    Shadow bytes around the buggy address:
      0x0c08800bab50: fa fa fd fd fd fd fd fa fa fa 00 00 00 00 00 00
      0x0c08800bab60: fa fa fd fd fd fd fd fa fa fa 00 00 00 00 00 fa
      0x0c08800bab70: fa fa 00 00 00 00 00 00 fa fa 00 00 00 00 00 00
      0x0c08800bab80: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fa
      0x0c08800bab90: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
    =>0x0c08800baba0: fa fa fd fd[fd]fd fd fa fa fa fd fd fd fd fd fa
      0x0c08800babb0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fa
      0x0c08800babc0: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
      0x0c08800babd0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fa
      0x0c08800babe0: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
      0x0c08800babf0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd
    Shadow byte legend (one shadow byte represents 8 application bytes):
      Addressable:           00
      Partially addressable: 01 02 03 04 05 06 07
      Heap left redzone:       fa
      Freed heap region:       fd
      Stack left redzone:      f1
      Stack mid redzone:       f2
      Stack right redzone:     f3
      Stack after return:      f5
      Stack use after scope:   f8
      Global redzone:          f9
      Global init order:       f6
      Poisoned by user:        f7
      Container overflow:      fc
      Array cookie:            ac
      Intra object redzone:    bb
      ASan internal:           fe
      Left alloca redzone:     ca
      Right alloca redzone:    cb
    ==24679==ABORTING

</details>
2021-10-05 21:07:10 +03:00
Kruglov Pavel
fa81606d6a
Fix build 2021-10-05 20:11:21 +03:00
Nikita Mikhaylov
cc150b7316
Merge pull request #29596 from nikitamikhaylov/fuzz
Add fuzzer for `executeQuery` function
2021-10-05 20:10:13 +03:00
Maksim Kita
4fde987ad1 Parse query from metadata exception throw fix 2021-10-05 19:34:25 +03:00
Anton Popov
b5098301d6 fix build 2021-10-05 18:23:53 +03:00
Mike Kot
25a62b9f83 Renaming TL to Typelist 2021-10-05 17:22:02 +02:00
Nikita Mikhaylov
a880cce0de better 2021-10-05 13:26:29 +00:00
kssenii
51f78e4446 Merge branch 'update-minio' of github.com:kssenii/ClickHouse into update-minio 2021-10-05 13:20:07 +00:00
kssenii
1c37b0df53 Ping tests 2021-10-05 13:19:43 +00:00
Filatenkov Artur
dc657a2c5e
Add error for multiple keys without current_key_id (#29546)
* add error for multiple keys without current_key_id

* add exception if one key with non zero id is used without current_key_id

* refactoring
2021-10-05 14:37:54 +03:00
Nikita Mikhaylov
f3c50e563e -nostdlib++ 2021-10-05 10:51:49 +00:00
alesapin
9aab0eb0a2 Fixup 2021-10-05 13:41:09 +03:00
mergify[bot]
fecb16c198
Merge branch 'master' into update-minio 2021-10-05 10:25:52 +00:00
alesapin
64158c011b Remove logging 2021-10-05 13:12:31 +03:00
Kruglov Pavel
e990f35ba3
Merge pull request #29746 from azat/JSONExtract-Null-fix
Forbid Nullable for JSONExtract* (JSONExtract() still supports Nullable), leads to SIGSEGV before
2021-10-05 13:04:46 +03:00
alesapin
c2f34928e0 Fixup 2021-10-05 13:04:44 +03:00
alesapin
3a8126372c Remove redundant header 2021-10-05 12:19:04 +03:00
alesapin
8186339307 Less seeks in compressed buffers 2021-10-05 12:11:25 +03:00
avogar
c2cb9e2721 Fix use of unititialized value 2021-10-05 11:55:26 +03:00
Kseniia Sumarokova
ead99b5911
Update docker_compose_minio.yml 2021-10-05 11:39:08 +03:00
Kseniia Sumarokova
8e03e0127a
Merge pull request #29737 from kssenii/fix-local
Better exception message for local interactive
2021-10-05 11:34:03 +03:00
Kseniia Sumarokova
438a981e76
Update TreeRewriter.cpp 2021-10-05 11:25:50 +03:00
kssenii
4962677a8b Update minio 2021-10-05 08:18:16 +00:00
Kseniia Sumarokova
385bf579a6
Merge pull request #29745 from azat/clickhouse-local-fix-stage
Fix --stage for clickhouse-local
2021-10-05 11:14:27 +03:00
Roman Bug
491c9c2065
Update docs/en/interfaces/formats.md
Co-authored-by: olgarev <56617294+olgarev@users.noreply.github.com>
2021-10-05 11:03:17 +03:00
Roman Bug
bf049015a3
Update docs/ru/interfaces/formats.md
Co-authored-by: olgarev <56617294+olgarev@users.noreply.github.com>
2021-10-05 11:03:11 +03:00
Roman Bug
9864655c45
Update docs/ru/interfaces/formats.md
Co-authored-by: olgarev <56617294+olgarev@users.noreply.github.com>
2021-10-05 11:02:37 +03:00
Roman Bug
3906fc9374
Update docs/en/interfaces/formats.md
Co-authored-by: olgarev <56617294+olgarev@users.noreply.github.com>
2021-10-05 11:02:30 +03:00
Roman Bug
f469d75b91
Update docs/en/interfaces/formats.md
Co-authored-by: olgarev <56617294+olgarev@users.noreply.github.com>
2021-10-05 11:02:19 +03:00
Vitaly Baranov
8a01b32cba
Merge pull request #28637 from vitlibar/fix-materialized-column-as-sharding-key
Fix materialized column as sharding key
2021-10-05 10:53:24 +03:00
alexey-milovidov
dab4c60596
Merge pull request #29757 from ClickHouse/changelog-21.10
Add changelog for 21.10
2021-10-05 09:31:16 +03:00
Alexey Milovidov
73665256b3 Add changelog for 21.10 2021-10-05 09:28:31 +03:00
Maksim Kita
3834256abf
Merge pull request #29722 from kitaisreal/borrowed-object-pool-condition-variable-notify-fix
BorrowedObjectPool condition variable notify fix
2021-10-04 21:39:59 +03:00
Maksim Kita
6a6946018b
Merge pull request #29747 from kitaisreal/table-function-dictionary-fix-comment
TableFunctionDictionary fix comment
2021-10-04 21:10:20 +03:00
Maksim Kita
6df81d94bf TableFunctionDictionary fix comment 2021-10-04 21:09:32 +03:00
Azat Khuzhin
20e706766c Forbid Nullable for JSONExtract* (JSONExtract() still supports Nullable)
Only JSONExtract() can support Nullable, others JSONExtract*
(JSONExtractString and similar) does not.

And right now this file is pretty complex already,
so adding support of Nullable for others will make it even more complex.

CI: https://clickhouse-test-reports.s3.yandex.net/29680/d0fc26f91a0141b56a0550741219c3dc43630e03/fuzzer_ubsan/report.html#fail1
2021-10-04 21:08:02 +03:00
Azat Khuzhin
bbee102e58 Fix --stage for clickhouse-local
This also fixes UBsan error, since query_processing_stage was not
initialized before for clickhouse-local.
2021-10-04 21:00:48 +03:00
Nikita Mikhaylov
ec2b30703e Better dictionary 2021-10-04 17:57:58 +00:00
Nikita Mikhaylov
ec99357097 Done 2021-10-04 17:57:58 +00:00
Nikita Mikhaylov
613b814e24
Merge pull request #29595 from CurtizJ/generalize-update-permutation
Generalize code in `IColumn::updatePermutation`
2021-10-04 20:36:57 +03:00
Nikita Mikhaylov
b31d11478b
Merge pull request #29680 from azat/defines
Cleanup common defines
2021-10-04 20:13:29 +03:00
Nikita Mikhaylov
6d5e9a9089
Merge pull request #29692 from azat/disk-memory-data-race-fix
Fix data-race in WriteIndirectBuffer (used in DiskMemory)
2021-10-04 20:13:18 +03:00
Nikita Mikhaylov
36516dd503
Merge pull request #29681 from azat/MemoryTracker-mutate-merge
Apply max_untracked_memory/memory_profiler_step/memory_profiler_sample_probability during mutate/merge
2021-10-04 20:04:57 +03:00