Commit Graph

73964 Commits

Author SHA1 Message Date
Azat Khuzhin
342ec02664 Fix concurrent access to LowCardinality during GROUP BY (leads to SIGSEGV)
The problem is that GROUP BY can update saved_hash, which can be also
updated by subsequent update of a dictionary, and this will lead to
use-after-free.

You will find ASan report in `details`.

<details>

    ==24679==ERROR: AddressSanitizer: heap-use-after-free on address 0x604000615d20 at pc 0x000022cc8684 bp 0x7ffea6b5f850 sp 0x7ffea6b5f848
    READ of size 8 at 0x604000615d20 thread T223 (QueryPipelineEx)
        0 0x22cc8683 in DB::ReverseIndex<>::insert(StringRef const&) obj-x86_64-linux-gnu/../src/Columns/ReverseIndex.h
        1 0x22cc0de1 in COW<DB::IColumn>::mutable_ptr<DB::IColumn> DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeImpl<char8_t>()::'lambda'()::operator()() const obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:540:39
        2 0x22cc0de1 in COW<DB::IColumn>::mutable_ptr<DB::IColumn> DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeImpl<char8_t>() obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:570:23
        3 0x22cb9c66 in COW<DB::IColumn>::mutable_ptr<DB::IColumn> DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeFrom()::'lambda'(auto)::operator()<char8_t>(auto) const obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:592:26
        4 0x22cb9c66 in DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeFrom() obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:600:28
        5 0x2500b897 in DB::ColumnLowCardinality::insertRangeFrom() obj-x86_64-linux-gnu/../src/Columns/ColumnLowCardinality.cpp:205:62
        6 0x25a182f4 in DB::appendBlock(DB::Block const&, DB::Block&) obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:470:23
        7 0x25a182f4 in DB::BufferSink::insertIntoBuffer(DB::Block const&, DB::StorageBuffer::Buffer&) obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:634:9
        8 0x25a173cc in DB::BufferSink::consume(DB::Chunk) obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:595:9
        9 0x26d1c997 in DB::SinkToStorage::transform(DB::Chunk&) obj-x86_64-linux-gnu/../src/Processors/Sinks/SinkToStorage.cpp:18:5

    0x604000615d20 is located 16 bytes inside of 40-byte region [0x604000615d10,0x604000615d38)
    freed by thread T37 (QueryPipelineEx) here:
        2 0x22cb9392 in boost::intrusive_ptr<DB::ColumnVector<unsigned long> >::~intrusive_ptr() obj-x86_64-linux-gnu/../contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:98:23
        4 0x22cb9392 in COW<DB::IColumn>::mutable_ptr<DB::ColumnVector<unsigned long> >::operator=() obj-x86_64-linux-gnu/../src/Common/COW.h💯57
        5 0x22cb9392 in DB::ReverseIndex<>::tryGetSavedHash() const obj-x86_64-linux-gnu/../src/Columns/ReverseIndex.h:362:28
        6 0x22cb9392 in DB::ColumnUnique<DB::ColumnString>::tryGetSavedHash() const obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:125:76
        7 0x242eaed3 in DB::ColumnsHashing::HashMethodSingleLowCardinalityColumn<>::HashMethodSingleLowCardinalityColumn() obj-x86_64-linux-gnu/../src/Common/ColumnsHashing.h:287:50
        8 0x242206c6 in void DB::Aggregator::executeImpl<>() const obj-x86_64-linux-gnu/../src/Interpreters/Aggregator.cpp:596:28
        9 0x24148e99 in DB::Aggregator::executeOnBlock() const obj-x86_64-linux-gnu/../src/Interpreters/Aggregator.cpp:1004:9
        10 0x26c24f3a in DB::AggregatingTransform::consume(DB::Chunk) obj-x86_64-linux-gnu/../src/Processors/Transforms/AggregatingTransform.cpp:539:33
        11 0x26c2054e in DB::AggregatingTransform::work() obj-x86_64-linux-gnu/../src/Processors/Transforms/AggregatingTransform.cpp:500:9

    previously allocated by thread T37 (QueryPipelineEx) here:
        0 0xb6d44fd in operator new(unsigned long) (/src/ch/tmp/upstream/clickhouse-asan+0xb6d44fd)
        1 0x11b78580 in COW<DB::IColumn>::mutable_ptr<> COWHelper<>::create<unsigned long&>(unsigned long&) (/src/ch/tmp/upstream/clickhouse-asan+0x11b78580)
        2 0x22cbf7b1 in DB::ReverseIndex<>::calcHashes() const obj-x86_64-linux-gnu/../src/Columns/ReverseIndex.h:472:17
        3 0x22cc2307 in DB::ReverseIndex<>::buildIndex() obj-x86_64-linux-gnu/../src/Columns/ReverseIndex.h:438:22
        4 0x22cc658c in DB::ReverseIndex<>::insert(StringRef const&) obj-x86_64-linux-gnu/../src/Columns/ReverseIndex.h:484:9
        5 0x22cc0de1 in COW<DB::IColumn>::mutable_ptr<DB::IColumn> DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeImpl<char8_t>()::'lambda'()::operator()() const obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:540:39
        6 0x22cc0de1 in COW<DB::IColumn>::mutable_ptr<DB::IColumn> DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeImpl<char8_t>() obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:570:23
        7 0x22cb9c66 in COW<DB::IColumn>::mutable_ptr<DB::IColumn> DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeFrom()::'lambda'(auto)::operator()<char8_t>(auto) const obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:592:26
        8 0x22cb9c66 in DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeFrom() obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:600:28
        9 0x2500b897 in DB::ColumnLowCardinality::insertRangeFrom() obj-x86_64-linux-gnu/../src/Columns/ColumnLowCardinality.cpp:205:62
        10 0x25a182f4 in DB::appendBlock(DB::Block const&, DB::Block&) obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:470:23
        11 0x25a182f4 in DB::BufferSink::insertIntoBuffer(DB::Block const&, DB::StorageBuffer::Buffer&) obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:634:9
        12 0x25a173cc in DB::BufferSink::consume(DB::Chunk) obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:595:9
        13 0x26d1c997 in DB::SinkToStorage::transform(DB::Chunk&) obj-x86_64-linux-gnu/../src/Processors/Sinks/SinkToStorage.cpp:18:5

    SUMMARY: AddressSanitizer: heap-use-after-free obj-x86_64-linux-gnu/../src/Columns/ReverseIndex.h in DB::ReverseIndex<unsigned long, DB::ColumnString>::insert(StringRef const&)
    Shadow bytes around the buggy address:
      0x0c08800bab50: fa fa fd fd fd fd fd fa fa fa 00 00 00 00 00 00
      0x0c08800bab60: fa fa fd fd fd fd fd fa fa fa 00 00 00 00 00 fa
      0x0c08800bab70: fa fa 00 00 00 00 00 00 fa fa 00 00 00 00 00 00
      0x0c08800bab80: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fa
      0x0c08800bab90: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
    =>0x0c08800baba0: fa fa fd fd[fd]fd fd fa fa fa fd fd fd fd fd fa
      0x0c08800babb0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fa
      0x0c08800babc0: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
      0x0c08800babd0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fa
      0x0c08800babe0: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
      0x0c08800babf0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd
    Shadow byte legend (one shadow byte represents 8 application bytes):
      Addressable:           00
      Partially addressable: 01 02 03 04 05 06 07
      Heap left redzone:       fa
      Freed heap region:       fd
      Stack left redzone:      f1
      Stack mid redzone:       f2
      Stack right redzone:     f3
      Stack after return:      f5
      Stack use after scope:   f8
      Global redzone:          f9
      Global init order:       f6
      Poisoned by user:        f7
      Container overflow:      fc
      Array cookie:            ac
      Intra object redzone:    bb
      ASan internal:           fe
      Left alloca redzone:     ca
      Right alloca redzone:    cb
    ==24679==ABORTING

</details>
2021-10-05 21:07:10 +03:00
Nikita Mikhaylov
cc150b7316
Merge pull request #29596 from nikitamikhaylov/fuzz
Add fuzzer for `executeQuery` function
2021-10-05 20:10:13 +03:00
Nikita Mikhaylov
a880cce0de better 2021-10-05 13:26:29 +00:00
Filatenkov Artur
dc657a2c5e
Add error for multiple keys without current_key_id (#29546)
* add error for multiple keys without current_key_id

* add exception if one key with non zero id is used without current_key_id

* refactoring
2021-10-05 14:37:54 +03:00
Nikita Mikhaylov
f3c50e563e -nostdlib++ 2021-10-05 10:51:49 +00:00
Kruglov Pavel
e990f35ba3
Merge pull request #29746 from azat/JSONExtract-Null-fix
Forbid Nullable for JSONExtract* (JSONExtract() still supports Nullable), leads to SIGSEGV before
2021-10-05 13:04:46 +03:00
Kseniia Sumarokova
8e03e0127a
Merge pull request #29737 from kssenii/fix-local
Better exception message for local interactive
2021-10-05 11:34:03 +03:00
Kseniia Sumarokova
385bf579a6
Merge pull request #29745 from azat/clickhouse-local-fix-stage
Fix --stage for clickhouse-local
2021-10-05 11:14:27 +03:00
Vitaly Baranov
8a01b32cba
Merge pull request #28637 from vitlibar/fix-materialized-column-as-sharding-key
Fix materialized column as sharding key
2021-10-05 10:53:24 +03:00
alexey-milovidov
dab4c60596
Merge pull request #29757 from ClickHouse/changelog-21.10
Add changelog for 21.10
2021-10-05 09:31:16 +03:00
Alexey Milovidov
73665256b3 Add changelog for 21.10 2021-10-05 09:28:31 +03:00
Maksim Kita
3834256abf
Merge pull request #29722 from kitaisreal/borrowed-object-pool-condition-variable-notify-fix
BorrowedObjectPool condition variable notify fix
2021-10-04 21:39:59 +03:00
Maksim Kita
6a6946018b
Merge pull request #29747 from kitaisreal/table-function-dictionary-fix-comment
TableFunctionDictionary fix comment
2021-10-04 21:10:20 +03:00
Maksim Kita
6df81d94bf TableFunctionDictionary fix comment 2021-10-04 21:09:32 +03:00
Azat Khuzhin
20e706766c Forbid Nullable for JSONExtract* (JSONExtract() still supports Nullable)
Only JSONExtract() can support Nullable, others JSONExtract*
(JSONExtractString and similar) does not.

And right now this file is pretty complex already,
so adding support of Nullable for others will make it even more complex.

CI: https://clickhouse-test-reports.s3.yandex.net/29680/d0fc26f91a0141b56a0550741219c3dc43630e03/fuzzer_ubsan/report.html#fail1
2021-10-04 21:08:02 +03:00
Azat Khuzhin
bbee102e58 Fix --stage for clickhouse-local
This also fixes UBsan error, since query_processing_stage was not
initialized before for clickhouse-local.
2021-10-04 21:00:48 +03:00
Nikita Mikhaylov
ec2b30703e Better dictionary 2021-10-04 17:57:58 +00:00
Nikita Mikhaylov
ec99357097 Done 2021-10-04 17:57:58 +00:00
Nikita Mikhaylov
613b814e24
Merge pull request #29595 from CurtizJ/generalize-update-permutation
Generalize code in `IColumn::updatePermutation`
2021-10-04 20:36:57 +03:00
Nikita Mikhaylov
b31d11478b
Merge pull request #29680 from azat/defines
Cleanup common defines
2021-10-04 20:13:29 +03:00
Nikita Mikhaylov
6d5e9a9089
Merge pull request #29692 from azat/disk-memory-data-race-fix
Fix data-race in WriteIndirectBuffer (used in DiskMemory)
2021-10-04 20:13:18 +03:00
Nikita Mikhaylov
36516dd503
Merge pull request #29681 from azat/MemoryTracker-mutate-merge
Apply max_untracked_memory/memory_profiler_step/memory_profiler_sample_probability during mutate/merge
2021-10-04 20:04:57 +03:00
Nikita Mikhaylov
731d6c5755
Merge pull request #29513 from azat/fuzzer-fix-server-pid
Fix server pid (hence exit code and attaching with gdb) in fuzzer tests
2021-10-04 19:59:24 +03:00
Nikolay Degterinsky
e2551e8cb0
Merge pull request #29490 from mehrshadhosseini28/patch-1
Update nlp-functions.md
2021-10-04 18:37:18 +03:00
kssenii
06fe822c22 Fix 2021-10-04 15:23:08 +00:00
Nikolai Kochetov
122d39cbba
Fix missing condition in pushed down predicate. (#29625) 2021-10-04 17:50:03 +03:00
Kseniia Sumarokova
d235520b3b
Merge pull request #29706 from kssenii/fix-flacky-test
Fix flacky test
2021-10-04 17:43:51 +03:00
Ivan Blinkov
dd2c4c2c3b
Update README.md 2021-10-04 17:12:29 +03:00
Anton Popov
362e84a336
Merge pull request #29429 from Enmk/Governance/database_comment
Implemented creating databases with comments
2021-10-04 15:12:33 +03:00
Maksim Kita
26611ba72c BorrowedObjectPool condition variable notify fix 2021-10-04 13:25:44 +03:00
Maksim Kita
8eda58acfb
Merge pull request #29516 from lingtaolf/feature/support-HSTS
Support HSTS in Clickhouse HTTP server
2021-10-04 12:49:51 +03:00
Nikolay Degterinsky
edf97d4736
Merge pull request #29718 from ne1r0n/patch-2
Fix typo in docs
2021-10-04 12:49:42 +03:00
Nikolay Degterinsky
719b8c213d
Merge pull request #29719 from ne1r0n/patch-3
Fix typo in docs
2021-10-04 12:49:18 +03:00
Nikolay Degterinsky
345ac59f46
Merge pull request #29552 from MaxWk/conversion-between-date32-and-datetime64
Support conversion between date32 and datetime64
2021-10-04 12:43:56 +03:00
Alexander Chashnikov
79294b6658
typo 2021-10-04 12:43:52 +03:00
Alexander Chashnikov
e76e4bdfd2
typo 2021-10-04 12:42:55 +03:00
Kseniia Sumarokova
a8f872f63e
Update test.py 2021-10-04 11:38:38 +03:00
Vitaly Baranov
17fe76709f
Merge pull request #28331 from vitlibar/mysql-authentication-cleanup
Clean up MySQL authentication.
2021-10-04 11:11:45 +03:00
Vitaly Baranov
1636ee24bb Fix using materialized column as sharding key. 2021-10-04 10:56:42 +03:00
Vitaly Baranov
217659c1c6 Add test. 2021-10-04 10:56:38 +03:00
Maksim Kita
25d1588d0a
Merge pull request #29666 from kitaisreal/user-defined-executable-functions-added-implicit-cast
UserDefinedExecutableFunctions added implicit cast
2021-10-04 10:34:31 +03:00
kssenii
a3ee621d5e Fix 2021-10-04 07:28:58 +00:00
Kseniia Sumarokova
5ab17d795f
Merge pull request #29673 from ClickHouse/url-fix-initialization
Fix initialization of `StorageURLSource`
2021-10-04 10:23:52 +03:00
Kseniia Sumarokova
1e1e3a6ad8
Merge pull request #29156 from sevirov/sevirov-DOCSUP-14334-clickhouse_over_web_server_with_static_files
DOCSUP-14334: ClickHouse over web server with static files
2021-10-04 09:48:55 +03:00
Kseniia Sumarokova
b8d73b291a
Merge pull request #29604 from olgarev/revolg-DOCSUP-14539-edit_and_translate
DOCSUP-14539: Edit and translate to Russian (MaterializedPostgreSQL settings)
2021-10-04 09:47:56 +03:00
Kseniia Sumarokova
0b2dd42261
Merge pull request #29686 from kssenii/fix-materialized-postgresql-3
MaterializedPostgreSQL fix
2021-10-04 09:47:21 +03:00
alexey-milovidov
325a01f798
Merge pull request #29553 from azat/group-by-with-totals-having-fix
Fix null deference for GROUP BY WITH TOTALS HAVING (when the column from HAVING wasn't selected)
2021-10-04 09:25:35 +03:00
Ivan Blinkov
1c55fd52b7
Sync copyrights 2021-10-04 09:16:18 +03:00
alexey-milovidov
6abb3d4c6d
Merge pull request #29682 from traceon/xcode-13-fix-build
Fix Xcode 13 build
2021-10-04 02:34:43 +03:00
alexey-milovidov
c410107994
Update CMakeLists.txt 2021-10-04 02:34:23 +03:00