Commit Graph

73965 Commits

Author SHA1 Message Date
Azat Khuzhin
a62728c16e Mark 02046_low_cardinality_parallel_group_by as long 2021-10-06 21:11:59 +03:00
Azat Khuzhin
342ec02664 Fix concurrent access to LowCardinality during GROUP BY (leads to SIGSEGV)
The problem is that GROUP BY can update saved_hash, which can be also
updated by subsequent update of a dictionary, and this will lead to
use-after-free.

You will find ASan report in `details`.

<details>

    ==24679==ERROR: AddressSanitizer: heap-use-after-free on address 0x604000615d20 at pc 0x000022cc8684 bp 0x7ffea6b5f850 sp 0x7ffea6b5f848
    READ of size 8 at 0x604000615d20 thread T223 (QueryPipelineEx)
        0 0x22cc8683 in DB::ReverseIndex<>::insert(StringRef const&) obj-x86_64-linux-gnu/../src/Columns/ReverseIndex.h
        1 0x22cc0de1 in COW<DB::IColumn>::mutable_ptr<DB::IColumn> DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeImpl<char8_t>()::'lambda'()::operator()() const obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:540:39
        2 0x22cc0de1 in COW<DB::IColumn>::mutable_ptr<DB::IColumn> DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeImpl<char8_t>() obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:570:23
        3 0x22cb9c66 in COW<DB::IColumn>::mutable_ptr<DB::IColumn> DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeFrom()::'lambda'(auto)::operator()<char8_t>(auto) const obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:592:26
        4 0x22cb9c66 in DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeFrom() obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:600:28
        5 0x2500b897 in DB::ColumnLowCardinality::insertRangeFrom() obj-x86_64-linux-gnu/../src/Columns/ColumnLowCardinality.cpp:205:62
        6 0x25a182f4 in DB::appendBlock(DB::Block const&, DB::Block&) obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:470:23
        7 0x25a182f4 in DB::BufferSink::insertIntoBuffer(DB::Block const&, DB::StorageBuffer::Buffer&) obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:634:9
        8 0x25a173cc in DB::BufferSink::consume(DB::Chunk) obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:595:9
        9 0x26d1c997 in DB::SinkToStorage::transform(DB::Chunk&) obj-x86_64-linux-gnu/../src/Processors/Sinks/SinkToStorage.cpp:18:5

    0x604000615d20 is located 16 bytes inside of 40-byte region [0x604000615d10,0x604000615d38)
    freed by thread T37 (QueryPipelineEx) here:
        2 0x22cb9392 in boost::intrusive_ptr<DB::ColumnVector<unsigned long> >::~intrusive_ptr() obj-x86_64-linux-gnu/../contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:98:23
        4 0x22cb9392 in COW<DB::IColumn>::mutable_ptr<DB::ColumnVector<unsigned long> >::operator=() obj-x86_64-linux-gnu/../src/Common/COW.h💯57
        5 0x22cb9392 in DB::ReverseIndex<>::tryGetSavedHash() const obj-x86_64-linux-gnu/../src/Columns/ReverseIndex.h:362:28
        6 0x22cb9392 in DB::ColumnUnique<DB::ColumnString>::tryGetSavedHash() const obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:125:76
        7 0x242eaed3 in DB::ColumnsHashing::HashMethodSingleLowCardinalityColumn<>::HashMethodSingleLowCardinalityColumn() obj-x86_64-linux-gnu/../src/Common/ColumnsHashing.h:287:50
        8 0x242206c6 in void DB::Aggregator::executeImpl<>() const obj-x86_64-linux-gnu/../src/Interpreters/Aggregator.cpp:596:28
        9 0x24148e99 in DB::Aggregator::executeOnBlock() const obj-x86_64-linux-gnu/../src/Interpreters/Aggregator.cpp:1004:9
        10 0x26c24f3a in DB::AggregatingTransform::consume(DB::Chunk) obj-x86_64-linux-gnu/../src/Processors/Transforms/AggregatingTransform.cpp:539:33
        11 0x26c2054e in DB::AggregatingTransform::work() obj-x86_64-linux-gnu/../src/Processors/Transforms/AggregatingTransform.cpp:500:9

    previously allocated by thread T37 (QueryPipelineEx) here:
        0 0xb6d44fd in operator new(unsigned long) (/src/ch/tmp/upstream/clickhouse-asan+0xb6d44fd)
        1 0x11b78580 in COW<DB::IColumn>::mutable_ptr<> COWHelper<>::create<unsigned long&>(unsigned long&) (/src/ch/tmp/upstream/clickhouse-asan+0x11b78580)
        2 0x22cbf7b1 in DB::ReverseIndex<>::calcHashes() const obj-x86_64-linux-gnu/../src/Columns/ReverseIndex.h:472:17
        3 0x22cc2307 in DB::ReverseIndex<>::buildIndex() obj-x86_64-linux-gnu/../src/Columns/ReverseIndex.h:438:22
        4 0x22cc658c in DB::ReverseIndex<>::insert(StringRef const&) obj-x86_64-linux-gnu/../src/Columns/ReverseIndex.h:484:9
        5 0x22cc0de1 in COW<DB::IColumn>::mutable_ptr<DB::IColumn> DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeImpl<char8_t>()::'lambda'()::operator()() const obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:540:39
        6 0x22cc0de1 in COW<DB::IColumn>::mutable_ptr<DB::IColumn> DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeImpl<char8_t>() obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:570:23
        7 0x22cb9c66 in COW<DB::IColumn>::mutable_ptr<DB::IColumn> DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeFrom()::'lambda'(auto)::operator()<char8_t>(auto) const obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:592:26
        8 0x22cb9c66 in DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeFrom() obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:600:28
        9 0x2500b897 in DB::ColumnLowCardinality::insertRangeFrom() obj-x86_64-linux-gnu/../src/Columns/ColumnLowCardinality.cpp:205:62
        10 0x25a182f4 in DB::appendBlock(DB::Block const&, DB::Block&) obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:470:23
        11 0x25a182f4 in DB::BufferSink::insertIntoBuffer(DB::Block const&, DB::StorageBuffer::Buffer&) obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:634:9
        12 0x25a173cc in DB::BufferSink::consume(DB::Chunk) obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:595:9
        13 0x26d1c997 in DB::SinkToStorage::transform(DB::Chunk&) obj-x86_64-linux-gnu/../src/Processors/Sinks/SinkToStorage.cpp:18:5

    SUMMARY: AddressSanitizer: heap-use-after-free obj-x86_64-linux-gnu/../src/Columns/ReverseIndex.h in DB::ReverseIndex<unsigned long, DB::ColumnString>::insert(StringRef const&)
    Shadow bytes around the buggy address:
      0x0c08800bab50: fa fa fd fd fd fd fd fa fa fa 00 00 00 00 00 00
      0x0c08800bab60: fa fa fd fd fd fd fd fa fa fa 00 00 00 00 00 fa
      0x0c08800bab70: fa fa 00 00 00 00 00 00 fa fa 00 00 00 00 00 00
      0x0c08800bab80: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fa
      0x0c08800bab90: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
    =>0x0c08800baba0: fa fa fd fd[fd]fd fd fa fa fa fd fd fd fd fd fa
      0x0c08800babb0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fa
      0x0c08800babc0: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
      0x0c08800babd0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fa
      0x0c08800babe0: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
      0x0c08800babf0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd
    Shadow byte legend (one shadow byte represents 8 application bytes):
      Addressable:           00
      Partially addressable: 01 02 03 04 05 06 07
      Heap left redzone:       fa
      Freed heap region:       fd
      Stack left redzone:      f1
      Stack mid redzone:       f2
      Stack right redzone:     f3
      Stack after return:      f5
      Stack use after scope:   f8
      Global redzone:          f9
      Global init order:       f6
      Poisoned by user:        f7
      Container overflow:      fc
      Array cookie:            ac
      Intra object redzone:    bb
      ASan internal:           fe
      Left alloca redzone:     ca
      Right alloca redzone:    cb
    ==24679==ABORTING

</details>
2021-10-05 21:07:10 +03:00
Nikita Mikhaylov
cc150b7316
Merge pull request #29596 from nikitamikhaylov/fuzz
Add fuzzer for `executeQuery` function
2021-10-05 20:10:13 +03:00
Nikita Mikhaylov
a880cce0de better 2021-10-05 13:26:29 +00:00
Filatenkov Artur
dc657a2c5e
Add error for multiple keys without current_key_id (#29546)
* add error for multiple keys without current_key_id

* add exception if one key with non zero id is used without current_key_id

* refactoring
2021-10-05 14:37:54 +03:00
Nikita Mikhaylov
f3c50e563e -nostdlib++ 2021-10-05 10:51:49 +00:00
Kruglov Pavel
e990f35ba3
Merge pull request #29746 from azat/JSONExtract-Null-fix
Forbid Nullable for JSONExtract* (JSONExtract() still supports Nullable), leads to SIGSEGV before
2021-10-05 13:04:46 +03:00
Kseniia Sumarokova
8e03e0127a
Merge pull request #29737 from kssenii/fix-local
Better exception message for local interactive
2021-10-05 11:34:03 +03:00
Kseniia Sumarokova
385bf579a6
Merge pull request #29745 from azat/clickhouse-local-fix-stage
Fix --stage for clickhouse-local
2021-10-05 11:14:27 +03:00
Vitaly Baranov
8a01b32cba
Merge pull request #28637 from vitlibar/fix-materialized-column-as-sharding-key
Fix materialized column as sharding key
2021-10-05 10:53:24 +03:00
alexey-milovidov
dab4c60596
Merge pull request #29757 from ClickHouse/changelog-21.10
Add changelog for 21.10
2021-10-05 09:31:16 +03:00
Alexey Milovidov
73665256b3 Add changelog for 21.10 2021-10-05 09:28:31 +03:00
Maksim Kita
3834256abf
Merge pull request #29722 from kitaisreal/borrowed-object-pool-condition-variable-notify-fix
BorrowedObjectPool condition variable notify fix
2021-10-04 21:39:59 +03:00
Maksim Kita
6a6946018b
Merge pull request #29747 from kitaisreal/table-function-dictionary-fix-comment
TableFunctionDictionary fix comment
2021-10-04 21:10:20 +03:00
Maksim Kita
6df81d94bf TableFunctionDictionary fix comment 2021-10-04 21:09:32 +03:00
Azat Khuzhin
20e706766c Forbid Nullable for JSONExtract* (JSONExtract() still supports Nullable)
Only JSONExtract() can support Nullable, others JSONExtract*
(JSONExtractString and similar) does not.

And right now this file is pretty complex already,
so adding support of Nullable for others will make it even more complex.

CI: https://clickhouse-test-reports.s3.yandex.net/29680/d0fc26f91a0141b56a0550741219c3dc43630e03/fuzzer_ubsan/report.html#fail1
2021-10-04 21:08:02 +03:00
Azat Khuzhin
bbee102e58 Fix --stage for clickhouse-local
This also fixes UBsan error, since query_processing_stage was not
initialized before for clickhouse-local.
2021-10-04 21:00:48 +03:00
Nikita Mikhaylov
ec2b30703e Better dictionary 2021-10-04 17:57:58 +00:00
Nikita Mikhaylov
ec99357097 Done 2021-10-04 17:57:58 +00:00
Nikita Mikhaylov
613b814e24
Merge pull request #29595 from CurtizJ/generalize-update-permutation
Generalize code in `IColumn::updatePermutation`
2021-10-04 20:36:57 +03:00
Nikita Mikhaylov
b31d11478b
Merge pull request #29680 from azat/defines
Cleanup common defines
2021-10-04 20:13:29 +03:00
Nikita Mikhaylov
6d5e9a9089
Merge pull request #29692 from azat/disk-memory-data-race-fix
Fix data-race in WriteIndirectBuffer (used in DiskMemory)
2021-10-04 20:13:18 +03:00
Nikita Mikhaylov
36516dd503
Merge pull request #29681 from azat/MemoryTracker-mutate-merge
Apply max_untracked_memory/memory_profiler_step/memory_profiler_sample_probability during mutate/merge
2021-10-04 20:04:57 +03:00
Nikita Mikhaylov
731d6c5755
Merge pull request #29513 from azat/fuzzer-fix-server-pid
Fix server pid (hence exit code and attaching with gdb) in fuzzer tests
2021-10-04 19:59:24 +03:00
Nikolay Degterinsky
e2551e8cb0
Merge pull request #29490 from mehrshadhosseini28/patch-1
Update nlp-functions.md
2021-10-04 18:37:18 +03:00
kssenii
06fe822c22 Fix 2021-10-04 15:23:08 +00:00
Nikolai Kochetov
122d39cbba
Fix missing condition in pushed down predicate. (#29625) 2021-10-04 17:50:03 +03:00
Kseniia Sumarokova
d235520b3b
Merge pull request #29706 from kssenii/fix-flacky-test
Fix flacky test
2021-10-04 17:43:51 +03:00
Ivan Blinkov
dd2c4c2c3b
Update README.md 2021-10-04 17:12:29 +03:00
Anton Popov
362e84a336
Merge pull request #29429 from Enmk/Governance/database_comment
Implemented creating databases with comments
2021-10-04 15:12:33 +03:00
Maksim Kita
26611ba72c BorrowedObjectPool condition variable notify fix 2021-10-04 13:25:44 +03:00
Maksim Kita
8eda58acfb
Merge pull request #29516 from lingtaolf/feature/support-HSTS
Support HSTS in Clickhouse HTTP server
2021-10-04 12:49:51 +03:00
Nikolay Degterinsky
edf97d4736
Merge pull request #29718 from ne1r0n/patch-2
Fix typo in docs
2021-10-04 12:49:42 +03:00
Nikolay Degterinsky
719b8c213d
Merge pull request #29719 from ne1r0n/patch-3
Fix typo in docs
2021-10-04 12:49:18 +03:00
Nikolay Degterinsky
345ac59f46
Merge pull request #29552 from MaxWk/conversion-between-date32-and-datetime64
Support conversion between date32 and datetime64
2021-10-04 12:43:56 +03:00
Alexander Chashnikov
79294b6658
typo 2021-10-04 12:43:52 +03:00
Alexander Chashnikov
e76e4bdfd2
typo 2021-10-04 12:42:55 +03:00
Kseniia Sumarokova
a8f872f63e
Update test.py 2021-10-04 11:38:38 +03:00
Vitaly Baranov
17fe76709f
Merge pull request #28331 from vitlibar/mysql-authentication-cleanup
Clean up MySQL authentication.
2021-10-04 11:11:45 +03:00
Vitaly Baranov
1636ee24bb Fix using materialized column as sharding key. 2021-10-04 10:56:42 +03:00
Vitaly Baranov
217659c1c6 Add test. 2021-10-04 10:56:38 +03:00
Maksim Kita
25d1588d0a
Merge pull request #29666 from kitaisreal/user-defined-executable-functions-added-implicit-cast
UserDefinedExecutableFunctions added implicit cast
2021-10-04 10:34:31 +03:00
kssenii
a3ee621d5e Fix 2021-10-04 07:28:58 +00:00
Kseniia Sumarokova
5ab17d795f
Merge pull request #29673 from ClickHouse/url-fix-initialization
Fix initialization of `StorageURLSource`
2021-10-04 10:23:52 +03:00
Kseniia Sumarokova
1e1e3a6ad8
Merge pull request #29156 from sevirov/sevirov-DOCSUP-14334-clickhouse_over_web_server_with_static_files
DOCSUP-14334: ClickHouse over web server with static files
2021-10-04 09:48:55 +03:00
Kseniia Sumarokova
b8d73b291a
Merge pull request #29604 from olgarev/revolg-DOCSUP-14539-edit_and_translate
DOCSUP-14539: Edit and translate to Russian (MaterializedPostgreSQL settings)
2021-10-04 09:47:56 +03:00
Kseniia Sumarokova
0b2dd42261
Merge pull request #29686 from kssenii/fix-materialized-postgresql-3
MaterializedPostgreSQL fix
2021-10-04 09:47:21 +03:00
alexey-milovidov
325a01f798
Merge pull request #29553 from azat/group-by-with-totals-having-fix
Fix null deference for GROUP BY WITH TOTALS HAVING (when the column from HAVING wasn't selected)
2021-10-04 09:25:35 +03:00
Ivan Blinkov
1c55fd52b7
Sync copyrights 2021-10-04 09:16:18 +03:00
alexey-milovidov
6abb3d4c6d
Merge pull request #29682 from traceon/xcode-13-fix-build
Fix Xcode 13 build
2021-10-04 02:34:43 +03:00