Commit Graph

74053 Commits

Author SHA1 Message Date
Nikolai Kochetov
25e2ebac75
Merge pull request #29782 from azat/lc-concurrent-GROUP-BY-fix
Fix concurrent access to LowCardinality during GROUP BY (leads to SIGSEGV)
2021-10-07 16:27:53 +03:00
tavplubix
754e038eec
Merge pull request #29803 from ClickHouse/intersects-previous-part-usability
Improve usability of error messages when error is caused by sophisticated interventions
2021-10-07 15:24:17 +03:00
Anton Popov
269a58373a
Merge pull request #29548 from ka1bi4/romanzhukov-DOCSUP-13988-JSONAsString-brackets
DOCSUP-13988: Allow data in square brackets in JSONAsString format
2021-10-07 14:50:07 +03:00
Anton Popov
4bc14dedfb
Merge pull request #28352 from Avogar/div-null
Avoid division by zero when denominator is Nullable
2021-10-07 14:47:53 +03:00
Kruglov Pavel
13b2fdc23b
Merge pull request #29659 from amosbird/jsonfix3
Fix nullable processing in JSONFunctions
2021-10-07 13:55:22 +03:00
Nikolai Kochetov
b8d3994765
Merge pull request #29829 from ClickHouse/revert-29827-coro-example
Revert "Add coroutines example."
2021-10-07 13:12:24 +03:00
Nikolai Kochetov
db79dc2a3d
Revert "Add coroutines example." 2021-10-07 13:11:51 +03:00
Nikolai Kochetov
8263d30321
Merge pull request #29827 from ClickHouse/coro-example
Add coroutines example.
2021-10-07 13:08:16 +03:00
Nikolai Kochetov
94a29545c8 Add logger. 2021-10-07 13:06:59 +03:00
Vladimir C
0422939893
Merge pull request #29281 from lingtaolf/bugfix/tupleFilter 2021-10-07 12:24:26 +03:00
Nikolai Kochetov
3e518861a0 Add coroutines example. 2021-10-07 12:23:22 +03:00
Nikolai Kochetov
4ab2e2745b
Merge pull request #29786 from azat/parallel_view_processing-fix
Fix parallel_view_processing
2021-10-07 12:15:26 +03:00
Nikolai Kochetov
78e1db209f
Remove more data streams (#29491)
* Remove more streams.

* Fixing build.

* Fixing build.

* Rename files.

* Fix fast test.

* Fix StorageKafka.

* Try fix kafka test.

* Move createBuffer to KafkaSource ctor.

* Revert "Move createBuffer to KafkaSource ctor."

This reverts commit 81fa94d27e.

* Revert "Try fix kafka test."

This reverts commit 2107e54969.

* Comment some rows in test.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2021-10-07 11:26:08 +03:00
Vladimir C
b02c12b182
Merge pull request #23881 from darkkeks/fix-cross-to-inline-join-error-condition 2021-10-07 11:09:36 +03:00
Vladimir C
4b80c6ff21
Merge pull request #29545 from CurtizJ/remove-permute 2021-10-07 11:07:34 +03:00
Maksim Kita
d91377c993
Merge pull request #29816 from kitaisreal/dictionary-attributes-updated-documentation
Dictionary attributes updated documentation
2021-10-07 10:21:15 +03:00
Maksim Kita
4ec7311d4d
Merge pull request #29738 from kitaisreal/added-function-ngram
Added function ngram
2021-10-07 10:21:00 +03:00
alexey-milovidov
16ad5953d6
Merge pull request #29783 from azat/tcphandler-progress-race
Fix data-race between fatal error handler and progress packets
2021-10-07 01:47:29 +03:00
avogar
568e27e9b3 Fix tests 2021-10-07 00:19:37 +03:00
Maksim Kita
c2c621c6dc Dictionary attributes updated documentation 2021-10-06 23:33:36 +03:00
Maksim Kita
b7b39da8cc Fixed build 2021-10-06 23:22:29 +03:00
Amos Bird
41a80a8852
Add more tests 2021-10-07 04:02:44 +08:00
Maksim Kita
d7dd593284
Merge pull request #29813 from azat/listen_backlog-config-default
Add listen_backlog to documentation
2021-10-06 22:20:04 +03:00
Azat Khuzhin
1c417be72e Apply suggestions from @kitaisreal 2021-10-06 22:19:09 +03:00
Maksim Kita
dce50c8b8a
Merge pull request #29779 from kitaisreal/parse-query-from-metadata-exception-throw-fix
Parse query from metadata exception throw fix
2021-10-06 21:29:12 +03:00
Amos Bird
a34a08268b
Fix nullable processing in JSONFunctions 2021-10-07 02:26:50 +08:00
Amos Bird
a391b64f57
Revert "Merge pull request #29746 from azat/JSONExtract-Null-fix"
This reverts commit e990f35ba3, reversing
changes made to 8e03e0127a.
2021-10-07 02:26:32 +08:00
Azat Khuzhin
83c271f446 Move QueryPipelineBuilder::setMaxThreads() into the InterpreterInsertQuery 2021-10-06 21:25:37 +03:00
Azat Khuzhin
a62728c16e Mark 02046_low_cardinality_parallel_group_by as long 2021-10-06 21:11:59 +03:00
Azat Khuzhin
74f5b92420 Add listen_backlog to documentation
Follow-up for: #29643
2021-10-06 21:09:00 +03:00
Kseniia Sumarokova
e88005c6f6
Merge pull request #29762 from kssenii/update-minio
May be fix s3 tests
2021-10-06 16:33:54 +03:00
Alexey Milovidov
1455cfeddd Improve usability of error messages when error is caused by sophisticated interventions 2021-10-06 15:28:52 +03:00
Maksim Kita
33353be4ed Fixed tests 2021-10-06 15:12:07 +03:00
Filatenkov Artur
a9e8ba0c00
Correct behaviour with unknown methods (#29057)
* correct behaviour with unknown methods

* move check on earlier stage

* add test

* correct filter for handler

* Update HTTPServerConnection.cpp

* remove trying to get params

* correct filter for head requests

* Update HTTPHandlerFactory.h

* Update HTTPHandlerFactory.h

* Trigger Build
2021-10-06 14:43:05 +03:00
Maksim Kita
a9e357aaff
Merge pull request #29792 from ClickHouse/ui-show-bars
Web UI: render bars in table cells
2021-10-06 13:49:40 +03:00
Maksim Kita
d2b3f3bd11
Merge pull request #29796 from kitaisreal/function-reinterpet-as-improve-readability
Function reinterpretAs improve readability
2021-10-06 13:44:24 +03:00
avogar
5e4cda02b6 Fix build 2021-10-06 13:38:22 +03:00
Maksim Kita
490ef61a03 Function reinterpretAs improve readability 2021-10-06 11:43:11 +03:00
Ivan Blinkov
8e9df6d667
Update sample.md 2021-10-06 09:12:27 +03:00
Ivan Blinkov
f45880aa89
Update sample-by.md 2021-10-06 09:02:20 +03:00
mergify[bot]
6eec0d76a7
Merge branch 'master' into update-minio 2021-10-06 05:45:18 +00:00
Alexey Milovidov
b5230a5fe2 Web UI: render bars in table cells 2021-10-06 03:59:16 +03:00
Maksim Kita
90eb74c933 Fixed tests 2021-10-06 00:37:45 +03:00
Kseniia Sumarokova
3663d23e4d
Merge pull request #29731 from kssenii/exists-subquery
Support EXISTS(subquery)
2021-10-05 22:26:19 +03:00
Azat Khuzhin
ae5570e6d6 Improve parallel_view_processing test 2021-10-05 22:20:46 +03:00
Azat Khuzhin
99ebdbb16e Fix parallel_view_processing
Number of threads got lost during after QueryPipelineBuilder::addChains()
2021-10-05 22:20:46 +03:00
Anton Popov
199ac31b48
Merge pull request #29683 from myrrc/improvement/tl-to-common
Non-recursive implementation for type list and its functions
2021-10-05 21:49:15 +03:00
Azat Khuzhin
4f5118b85f Fix data-race between fatal error handler and progress packets
This is not that important, but because of packets overlaps the client
may not render the fatal error.

You will find TSan report in `details`.

<details>

```
WARNING: ThreadSanitizer: data race (pid=13384)
  Write of size 8 at 0x7b24000ad818 by thread T47:
    0 DB::WriteBuffer::next() obj-x86_64-linux-gnu/../src/IO/WriteBuffer.h:43:15 (clickhouse-tsan+0x16124d61)
    1 DB::TCPHandler::sendProgress() obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:1602:10 (clickhouse-tsan+0x16124d61)
    2 DB::TCPHandler::runImpl()::$_6::operator()() const obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:318:25 (clickhouse-tsan+0x1612cf54)

  Previous write of size 8 at 0x7b24000ad818 by thread T229:
    0 DB::WriteBuffer::next() obj-x86_64-linux-gnu/../src/IO/WriteBuffer.h:43:15 (clickhouse-tsan+0x1612b793)
    1 DB::TCPHandler::sendLogData(DB::Block const&) obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:1561:10 (clickhouse-tsan+0x1612b793)
    2 DB::TCPHandler::sendLogs() obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:1632:9 (clickhouse-tsan+0x161228ab)
    3 DB::TCPHandler::runImpl()::$_1::operator()() const obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:229:62 (clickhouse-tsan+0x1612c4e1)
```

</details>
2021-10-05 21:21:32 +03:00
Azat Khuzhin
342ec02664 Fix concurrent access to LowCardinality during GROUP BY (leads to SIGSEGV)
The problem is that GROUP BY can update saved_hash, which can be also
updated by subsequent update of a dictionary, and this will lead to
use-after-free.

You will find ASan report in `details`.

<details>

    ==24679==ERROR: AddressSanitizer: heap-use-after-free on address 0x604000615d20 at pc 0x000022cc8684 bp 0x7ffea6b5f850 sp 0x7ffea6b5f848
    READ of size 8 at 0x604000615d20 thread T223 (QueryPipelineEx)
        0 0x22cc8683 in DB::ReverseIndex<>::insert(StringRef const&) obj-x86_64-linux-gnu/../src/Columns/ReverseIndex.h
        1 0x22cc0de1 in COW<DB::IColumn>::mutable_ptr<DB::IColumn> DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeImpl<char8_t>()::'lambda'()::operator()() const obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:540:39
        2 0x22cc0de1 in COW<DB::IColumn>::mutable_ptr<DB::IColumn> DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeImpl<char8_t>() obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:570:23
        3 0x22cb9c66 in COW<DB::IColumn>::mutable_ptr<DB::IColumn> DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeFrom()::'lambda'(auto)::operator()<char8_t>(auto) const obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:592:26
        4 0x22cb9c66 in DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeFrom() obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:600:28
        5 0x2500b897 in DB::ColumnLowCardinality::insertRangeFrom() obj-x86_64-linux-gnu/../src/Columns/ColumnLowCardinality.cpp:205:62
        6 0x25a182f4 in DB::appendBlock(DB::Block const&, DB::Block&) obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:470:23
        7 0x25a182f4 in DB::BufferSink::insertIntoBuffer(DB::Block const&, DB::StorageBuffer::Buffer&) obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:634:9
        8 0x25a173cc in DB::BufferSink::consume(DB::Chunk) obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:595:9
        9 0x26d1c997 in DB::SinkToStorage::transform(DB::Chunk&) obj-x86_64-linux-gnu/../src/Processors/Sinks/SinkToStorage.cpp:18:5

    0x604000615d20 is located 16 bytes inside of 40-byte region [0x604000615d10,0x604000615d38)
    freed by thread T37 (QueryPipelineEx) here:
        2 0x22cb9392 in boost::intrusive_ptr<DB::ColumnVector<unsigned long> >::~intrusive_ptr() obj-x86_64-linux-gnu/../contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:98:23
        4 0x22cb9392 in COW<DB::IColumn>::mutable_ptr<DB::ColumnVector<unsigned long> >::operator=() obj-x86_64-linux-gnu/../src/Common/COW.h💯57
        5 0x22cb9392 in DB::ReverseIndex<>::tryGetSavedHash() const obj-x86_64-linux-gnu/../src/Columns/ReverseIndex.h:362:28
        6 0x22cb9392 in DB::ColumnUnique<DB::ColumnString>::tryGetSavedHash() const obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:125:76
        7 0x242eaed3 in DB::ColumnsHashing::HashMethodSingleLowCardinalityColumn<>::HashMethodSingleLowCardinalityColumn() obj-x86_64-linux-gnu/../src/Common/ColumnsHashing.h:287:50
        8 0x242206c6 in void DB::Aggregator::executeImpl<>() const obj-x86_64-linux-gnu/../src/Interpreters/Aggregator.cpp:596:28
        9 0x24148e99 in DB::Aggregator::executeOnBlock() const obj-x86_64-linux-gnu/../src/Interpreters/Aggregator.cpp:1004:9
        10 0x26c24f3a in DB::AggregatingTransform::consume(DB::Chunk) obj-x86_64-linux-gnu/../src/Processors/Transforms/AggregatingTransform.cpp:539:33
        11 0x26c2054e in DB::AggregatingTransform::work() obj-x86_64-linux-gnu/../src/Processors/Transforms/AggregatingTransform.cpp:500:9

    previously allocated by thread T37 (QueryPipelineEx) here:
        0 0xb6d44fd in operator new(unsigned long) (/src/ch/tmp/upstream/clickhouse-asan+0xb6d44fd)
        1 0x11b78580 in COW<DB::IColumn>::mutable_ptr<> COWHelper<>::create<unsigned long&>(unsigned long&) (/src/ch/tmp/upstream/clickhouse-asan+0x11b78580)
        2 0x22cbf7b1 in DB::ReverseIndex<>::calcHashes() const obj-x86_64-linux-gnu/../src/Columns/ReverseIndex.h:472:17
        3 0x22cc2307 in DB::ReverseIndex<>::buildIndex() obj-x86_64-linux-gnu/../src/Columns/ReverseIndex.h:438:22
        4 0x22cc658c in DB::ReverseIndex<>::insert(StringRef const&) obj-x86_64-linux-gnu/../src/Columns/ReverseIndex.h:484:9
        5 0x22cc0de1 in COW<DB::IColumn>::mutable_ptr<DB::IColumn> DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeImpl<char8_t>()::'lambda'()::operator()() const obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:540:39
        6 0x22cc0de1 in COW<DB::IColumn>::mutable_ptr<DB::IColumn> DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeImpl<char8_t>() obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:570:23
        7 0x22cb9c66 in COW<DB::IColumn>::mutable_ptr<DB::IColumn> DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeFrom()::'lambda'(auto)::operator()<char8_t>(auto) const obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:592:26
        8 0x22cb9c66 in DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeFrom() obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:600:28
        9 0x2500b897 in DB::ColumnLowCardinality::insertRangeFrom() obj-x86_64-linux-gnu/../src/Columns/ColumnLowCardinality.cpp:205:62
        10 0x25a182f4 in DB::appendBlock(DB::Block const&, DB::Block&) obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:470:23
        11 0x25a182f4 in DB::BufferSink::insertIntoBuffer(DB::Block const&, DB::StorageBuffer::Buffer&) obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:634:9
        12 0x25a173cc in DB::BufferSink::consume(DB::Chunk) obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:595:9
        13 0x26d1c997 in DB::SinkToStorage::transform(DB::Chunk&) obj-x86_64-linux-gnu/../src/Processors/Sinks/SinkToStorage.cpp:18:5

    SUMMARY: AddressSanitizer: heap-use-after-free obj-x86_64-linux-gnu/../src/Columns/ReverseIndex.h in DB::ReverseIndex<unsigned long, DB::ColumnString>::insert(StringRef const&)
    Shadow bytes around the buggy address:
      0x0c08800bab50: fa fa fd fd fd fd fd fa fa fa 00 00 00 00 00 00
      0x0c08800bab60: fa fa fd fd fd fd fd fa fa fa 00 00 00 00 00 fa
      0x0c08800bab70: fa fa 00 00 00 00 00 00 fa fa 00 00 00 00 00 00
      0x0c08800bab80: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fa
      0x0c08800bab90: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
    =>0x0c08800baba0: fa fa fd fd[fd]fd fd fa fa fa fd fd fd fd fd fa
      0x0c08800babb0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fa
      0x0c08800babc0: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
      0x0c08800babd0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fa
      0x0c08800babe0: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
      0x0c08800babf0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd
    Shadow byte legend (one shadow byte represents 8 application bytes):
      Addressable:           00
      Partially addressable: 01 02 03 04 05 06 07
      Heap left redzone:       fa
      Freed heap region:       fd
      Stack left redzone:      f1
      Stack mid redzone:       f2
      Stack right redzone:     f3
      Stack after return:      f5
      Stack use after scope:   f8
      Global redzone:          f9
      Global init order:       f6
      Poisoned by user:        f7
      Container overflow:      fc
      Array cookie:            ac
      Intra object redzone:    bb
      ASan internal:           fe
      Left alloca redzone:     ca
      Right alloca redzone:    cb
    ==24679==ABORTING

</details>
2021-10-05 21:07:10 +03:00
Kruglov Pavel
fa81606d6a
Fix build 2021-10-05 20:11:21 +03:00