thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-14 10:22:10 +00:00
Code Issues Packages Projects Releases Wiki Activity
84,118 Commits 9,328 Branches 1,783 Tags 2.1 GiB
9058e776e5
Commit Graph

560 Commits

Author SHA1 Message Date
jasperzhu
49dd69c6c9 adjust header files 2021-10-27 05:22:05 +05:30
jasperzhu
0217ed09ac revise compile flag 2021-10-27 05:19:15 +05:30
jasperzhu
5ab55f7eaa fixed function defect 2021-10-27 04:21:46 +05:30
jasperzhu
27f30dab89 remove function due to extra latency 2021-10-27 03:44:28 +05:30
jasperzhu
fb31410926 change to inline for bitmask generator 2021-10-27 00:05:47 +05:30
jasperzhu
1caff8a81c fixed compile error 2021-10-26 22:12:21 +05:30
jasperzhu
911b9c3973 Improve function defect 2021-10-26 06:02:57 +05:30
jasperzhu
3d115536f5 Improve filter bitmask generator function all in one with sse/avx2/avx512 2021-10-26 01:39:09 +05:30
Maksim Kita
243be5028a
Merge pull request #30014 from jinjunzh/master 
add x86 feature avx2/avx512 support for accelerating filter operation
 2021-10-22 14:16:47 +03:00
vesslanjin
f607a5d0ab remove branch in ColumnDecimal, same as pull req #29881 
Co-authored-by: Zhu Jasper jasper.zhu@intel.com

Signed-off-by: vesslanjin <jun.i.jin@intel.com>
 2021-10-20 07:40:06 -04:00
Anton Popov
d71ffc355a Merge remote-tracking branch 'upstream/master' into HEAD 2021-10-18 15:18:22 +03:00
Anton Popov
e46396331d better permute in ColumnSparse 2021-10-14 20:28:31 +03:00
Anton Popov
7aa6068fb2 Merge remote-tracking branch 'upstream/master' into HEAD 2021-10-14 19:44:08 +03:00
jasperzhu
f08dd0e8d8 modify flags definitions 2021-10-13 05:13:34 +05:30
jasperzhu
a1ab3b6606 Fixed code style issue 2021-10-12 22:44:37 +05:30
jasperzhu
8f349907fb Fixed trailing whitespaces issue 2021-10-12 09:21:54 +05:30
jasperzhu
3be83a26d0 fixed Typos and Styles issues 2021-10-12 08:24:47 +05:30
jasperzhu
0ef2624408 add specific flags for compiling filter operation source files 2021-10-12 04:04:27 +05:30
jasperzhu
a3d629a5b5 add x86 feature avx2/avx512 support for filter implementation 2021-10-11 22:51:13 +05:30
Nikolai Kochetov
a95c28ec4b
Merge pull request #29898 from ClickHouse/remove-native-stream 
Remove some more streams.
 2021-10-10 21:01:16 +03:00
Vladimir C
3940382c2f
Remove defined(__POPCNT__) from column filter 2021-10-09 13:34:02 +03:00
vesslanjin
bed09ee68d Whitespace fix 
Signed-off-by: vesslanjin <jun.i.jin@intel.com>
 2021-10-09 02:56:10 -04:00
vesslanjin
75af011068 use while instead of popcnt + for 
Co-authored-by: Zhu Jasper jasper.zhu@intel.com
 2021-10-08 14:58:23 -04:00
Nikolai Kochetov
340b53ef85 Remove some more streams. 2021-10-08 17:03:54 +03:00
Vladimir C
525df99893
Whitespace fix 2021-10-08 12:10:19 +03:00
Vladimir C
35feb08236
Whitespace changes 2021-10-08 12:03:55 +03:00
vesslanjin
383f3a3a20 Remove branchy code in filter operation with a better implementation with popcnt/ctz which have better performance 
Co-authored-by: Zhu Jasper <jasper.zhu@intel.com>
 2021-10-08 09:19:58 -04:00
Nikolai Kochetov
25e2ebac75
Merge pull request #29782 from azat/lc-concurrent-GROUP-BY-fix 
Fix concurrent access to LowCardinality during GROUP BY (leads to SIGSEGV)
 2021-10-07 16:27:53 +03:00
Vladimir C
4b80c6ff21
Merge pull request #29545 from CurtizJ/remove-permute 2021-10-07 11:07:34 +03:00
mergify[bot]
5b667d3cd5
Merge branch 'master' into fix_IP_conversion 2021-10-06 16:40:47 +00:00
Anton Popov
199ac31b48
Merge pull request #29683 from myrrc/improvement/tl-to-common 
Non-recursive implementation for type list and its functions
 2021-10-05 21:49:15 +03:00
Azat Khuzhin
342ec02664 Fix concurrent access to LowCardinality during GROUP BY (leads to SIGSEGV) 
The problem is that GROUP BY can update saved_hash, which can be also
updated by subsequent update of a dictionary, and this will lead to
use-after-free.

You will find ASan report in `details`.

<details>

    ==24679==ERROR: AddressSanitizer: heap-use-after-free on address 0x604000615d20 at pc 0x000022cc8684 bp 0x7ffea6b5f850 sp 0x7ffea6b5f848
    READ of size 8 at 0x604000615d20 thread T223 (QueryPipelineEx)
        0 0x22cc8683 in DB::ReverseIndex<>::insert(StringRef const&) obj-x86_64-linux-gnu/../src/Columns/ReverseIndex.h
        1 0x22cc0de1 in COW<DB::IColumn>::mutable_ptr<DB::IColumn> DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeImpl<char8_t>()::'lambda'()::operator()() const obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:540:39
        2 0x22cc0de1 in COW<DB::IColumn>::mutable_ptr<DB::IColumn> DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeImpl<char8_t>() obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:570:23
        3 0x22cb9c66 in COW<DB::IColumn>::mutable_ptr<DB::IColumn> DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeFrom()::'lambda'(auto)::operator()<char8_t>(auto) const obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:592:26
        4 0x22cb9c66 in DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeFrom() obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:600:28
        5 0x2500b897 in DB::ColumnLowCardinality::insertRangeFrom() obj-x86_64-linux-gnu/../src/Columns/ColumnLowCardinality.cpp:205:62
        6 0x25a182f4 in DB::appendBlock(DB::Block const&, DB::Block&) obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:470:23
        7 0x25a182f4 in DB::BufferSink::insertIntoBuffer(DB::Block const&, DB::StorageBuffer::Buffer&) obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:634:9
        8 0x25a173cc in DB::BufferSink::consume(DB::Chunk) obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:595:9
        9 0x26d1c997 in DB::SinkToStorage::transform(DB::Chunk&) obj-x86_64-linux-gnu/../src/Processors/Sinks/SinkToStorage.cpp:18:5

    0x604000615d20 is located 16 bytes inside of 40-byte region [0x604000615d10,0x604000615d38)
    freed by thread T37 (QueryPipelineEx) here:
        2 0x22cb9392 in boost::intrusive_ptr<DB::ColumnVector<unsigned long> >::~intrusive_ptr() obj-x86_64-linux-gnu/../contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:98:23
        4 0x22cb9392 in COW<DB::IColumn>::mutable_ptr<DB::ColumnVector<unsigned long> >::operator=() obj-x86_64-linux-gnu/../src/Common/COW.h💯57
        5 0x22cb9392 in DB::ReverseIndex<>::tryGetSavedHash() const obj-x86_64-linux-gnu/../src/Columns/ReverseIndex.h:362:28
        6 0x22cb9392 in DB::ColumnUnique<DB::ColumnString>::tryGetSavedHash() const obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:125:76
        7 0x242eaed3 in DB::ColumnsHashing::HashMethodSingleLowCardinalityColumn<>::HashMethodSingleLowCardinalityColumn() obj-x86_64-linux-gnu/../src/Common/ColumnsHashing.h:287:50
        8 0x242206c6 in void DB::Aggregator::executeImpl<>() const obj-x86_64-linux-gnu/../src/Interpreters/Aggregator.cpp:596:28
        9 0x24148e99 in DB::Aggregator::executeOnBlock() const obj-x86_64-linux-gnu/../src/Interpreters/Aggregator.cpp:1004:9
        10 0x26c24f3a in DB::AggregatingTransform::consume(DB::Chunk) obj-x86_64-linux-gnu/../src/Processors/Transforms/AggregatingTransform.cpp:539:33
        11 0x26c2054e in DB::AggregatingTransform::work() obj-x86_64-linux-gnu/../src/Processors/Transforms/AggregatingTransform.cpp:500:9

    previously allocated by thread T37 (QueryPipelineEx) here:
        0 0xb6d44fd in operator new(unsigned long) (/src/ch/tmp/upstream/clickhouse-asan+0xb6d44fd)
        1 0x11b78580 in COW<DB::IColumn>::mutable_ptr<> COWHelper<>::create<unsigned long&>(unsigned long&) (/src/ch/tmp/upstream/clickhouse-asan+0x11b78580)
        2 0x22cbf7b1 in DB::ReverseIndex<>::calcHashes() const obj-x86_64-linux-gnu/../src/Columns/ReverseIndex.h:472:17
        3 0x22cc2307 in DB::ReverseIndex<>::buildIndex() obj-x86_64-linux-gnu/../src/Columns/ReverseIndex.h:438:22
        4 0x22cc658c in DB::ReverseIndex<>::insert(StringRef const&) obj-x86_64-linux-gnu/../src/Columns/ReverseIndex.h:484:9
        5 0x22cc0de1 in COW<DB::IColumn>::mutable_ptr<DB::IColumn> DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeImpl<char8_t>()::'lambda'()::operator()() const obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:540:39
        6 0x22cc0de1 in COW<DB::IColumn>::mutable_ptr<DB::IColumn> DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeImpl<char8_t>() obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:570:23
        7 0x22cb9c66 in COW<DB::IColumn>::mutable_ptr<DB::IColumn> DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeFrom()::'lambda'(auto)::operator()<char8_t>(auto) const obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:592:26
        8 0x22cb9c66 in DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeFrom() obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:600:28
        9 0x2500b897 in DB::ColumnLowCardinality::insertRangeFrom() obj-x86_64-linux-gnu/../src/Columns/ColumnLowCardinality.cpp:205:62
        10 0x25a182f4 in DB::appendBlock(DB::Block const&, DB::Block&) obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:470:23
        11 0x25a182f4 in DB::BufferSink::insertIntoBuffer(DB::Block const&, DB::StorageBuffer::Buffer&) obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:634:9
        12 0x25a173cc in DB::BufferSink::consume(DB::Chunk) obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:595:9
        13 0x26d1c997 in DB::SinkToStorage::transform(DB::Chunk&) obj-x86_64-linux-gnu/../src/Processors/Sinks/SinkToStorage.cpp:18:5

    SUMMARY: AddressSanitizer: heap-use-after-free obj-x86_64-linux-gnu/../src/Columns/ReverseIndex.h in DB::ReverseIndex<unsigned long, DB::ColumnString>::insert(StringRef const&)
    Shadow bytes around the buggy address:
      0x0c08800bab50: fa fa fd fd fd fd fd fa fa fa 00 00 00 00 00 00
      0x0c08800bab60: fa fa fd fd fd fd fd fa fa fa 00 00 00 00 00 fa
      0x0c08800bab70: fa fa 00 00 00 00 00 00 fa fa 00 00 00 00 00 00
      0x0c08800bab80: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fa
      0x0c08800bab90: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
    =>0x0c08800baba0: fa fa fd fd[fd]fd fd fa fa fa fd fd fd fd fd fa
      0x0c08800babb0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fa
      0x0c08800babc0: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
      0x0c08800babd0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fa
      0x0c08800babe0: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
      0x0c08800babf0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd
    Shadow byte legend (one shadow byte represents 8 application bytes):
      Addressable:           00
      Partially addressable: 01 02 03 04 05 06 07
      Heap left redzone:       fa
      Freed heap region:       fd
      Stack left redzone:      f1
      Stack mid redzone:       f2
      Stack right redzone:     f3
      Stack after return:      f5
      Stack use after scope:   f8
      Global redzone:          f9
      Global init order:       f6
      Poisoned by user:        f7
      Container overflow:      fc
      Array cookie:            ac
      Intra object redzone:    bb
      ASan internal:           fe
      Left alloca redzone:     ca
      Right alloca redzone:    cb
    ==24679==ABORTING

</details>
 2021-10-05 21:07:10 +03:00
Anton Popov
b5098301d6 fix build 2021-10-05 18:23:53 +03:00
Anton Popov
cd08057285 better arguments 2021-10-04 15:02:11 +03:00
Mike Kot
8c8bada56e Non-recursive implementation for TL and its functions 2021-10-03 18:02:04 +02:00
Anton Popov
857b8f0b1d fix clang-tidy 2021-10-02 22:36:30 +03:00
Anton Popov
1f10246a10 Merge remote-tracking branch 'upstream/master' into HEAD 2021-10-02 22:33:27 +03:00
Alexey Milovidov
fe6b7c77c7 Rename "common" to "base" 2021-10-02 10:13:14 +03:00
Anton Popov
9c296e94f3 fix performance of updatePermutation 2021-10-01 19:45:17 +03:00
Anton Popov
515cc74530 generalize code in IColumn::updatePermutation 2021-10-01 00:11:54 +03:00
Anton Popov
0dfe96fc02 generalize code in IColumn::permute 2021-09-29 20:56:10 +03:00
Dmitry Novik
abdd8fb516 Cleanup 2021-09-29 19:42:41 +03:00
Anton Popov
914781052e Merge remote-tracking branch 'upstream/master' into HEAD 2021-09-29 17:37:07 +03:00
Anton Popov
dc94d2239e better ColumnSparse 2021-09-29 17:18:41 +03:00
Dmitry Novik
95193e06f6 Remove empty row creation 2021-09-28 16:59:58 +03:00
mergify[bot]
86eb589c72
Merge branch 'master' into fix_IP_conversion 2021-09-27 14:41:56 +00:00
alexey-milovidov
e4363872ca
Merge branch 'master' into yandex-to-clickhouse-in-configs 2021-09-25 19:58:36 +03:00
Alexey Milovidov
cd7f9d981c Remove ya.make 2021-09-25 04:22:54 +03:00
Alexey Milovidov
e9e77b4403 .tech -> .com 2021-09-22 03:22:57 +03:00
Dmitry Novik
9049fd701f Refactor ColumnStringHelpers 2021-09-21 17:27:34 +03:00
Vasily Nemkov
175b94fa2d Special case of conversion code for types with custom serialization/deserialization 
Fixes CASTing from String or FixedString to IPv4 or IPv6 and back.
 2021-09-21 15:34:06 +03:00
Anton Popov
eef436fe22 Merge remote-tracking branch 'upstream/master' into HEAD 2021-09-16 18:07:42 +03:00
Anton Popov
348537faee better IColumn::createWithOffsets 2021-09-16 16:57:45 +03:00
Anton Popov
3adc2a9aea fix distinct with sparse columns 2021-09-16 15:52:59 +03:00
Sergei Semin
12e24a23d3
Merge branch 'master' into fix-build-clang-13-2 2021-09-12 19:30:55 +03:00
alexey-milovidov
60f76d9254
Merge pull request #28759 from nikitamikhaylov/enable-gcc-11 
Switch CI to GCC-11
 2021-09-12 15:56:57 +03:00
alexey-milovidov
ea13a8b562
Merge pull request #28659 from myrrc/improvement/tostring_to_magic_enum 
Improving CH type system with concepts
 2021-09-12 15:26:29 +03:00
Sergei Semin
f74350c148 fix special build 2021-09-12 02:49:31 +03:00
Mike Kot
3401b32e37 Adapted usage of is_decimal concept 2021-09-11 19:35:12 +02:00
Pavel Kruglov
9c756fc4b2 Fix bug with LowCardinality in short-curcuit function evaluation 2021-09-11 12:56:25 +03:00
Mike Kot
bce011cb33 Replacing IsDecimalNumber 2021-09-11 00:40:18 +02:00
Nikita Mikhaylov
513d40feb2 Fix build 2021-09-09 14:59:25 +00:00
Anton Popov
8203bd1ac6 Merge remote-tracking branch 'upstream/master' into HEAD 2021-09-09 14:04:37 +03:00
Mike Kot
8e9aacadd1 Initial: replacing hardcoded toString for enums with magic_enum 2021-09-06 16:24:03 +02:00
Nikolai Kochetov
af0da33147
Merge pull request #27827 from ClickHouse/more-checks-for-lv-and-native 
Add more checks for LC in native protocol.
 2021-09-02 17:27:42 +03:00
alexey-milovidov
4cc0b0298c
Merge pull request #28269 from amosbird/fixweirdcode 
Better nullable primary key implementation
 2021-09-01 00:48:45 +03:00
Raúl Marín
7fb57482c0 Fix bug in countBytesInFilterWithNull 2021-08-30 13:35:22 +02:00
Dmitrii Kovalkov
9871ad70ff Exclude fuzzers 2021-08-30 11:12:25 +03:00
Amos Bird
f2374a6916
Better nullable primary key implementation. 2021-08-28 17:48:28 +08:00
Nikolai Kochetov
9c028be1e4 Fix insertion of fields with Infinity values in nullable low cardinality columns. 2021-08-26 21:19:12 +03:00
Vitaly Baranov
cbc1b2e72d Fix selecting with extremes from LowCardinality(UUID) 2021-08-20 13:10:41 +03:00
Anton Popov
c3c3a06078 Merge remote-tracking branch 'upstream/master' into HEAD 2021-08-20 01:45:38 +03:00
Nikolai Kochetov
094ad9468b Fix style. 2021-08-19 16:09:56 +03:00
Nikolai Kochetov
3e44ef437e
Merge pull request #26639 from ClickHouse/fix-bad-cast 
Fix bad cast
 2021-08-19 11:17:26 +03:00
Nikolai Kochetov
108137a67e Add more checks for LC in native protocol. 2021-08-18 19:15:57 +03:00
Pavel Kruglov
240398d58c Fix bug in expand() method 2021-08-18 19:13:07 +03:00
alexey-milovidov
d88cf81d71
Merge branch 'master' into fix-bad-cast 2021-08-15 09:06:42 +03:00
mergify[bot]
319e5af6eb
Merge branch 'master' into short-circuit 2021-08-13 13:01:46 +00:00
Pavel Kruglov
36bea6abb4 Add setting optimize_short_circuit_function_evaluation 2021-08-13 14:48:32 +03:00
Pavel Kruglov
fc59c98676 Clean up 2021-08-13 11:18:34 +03:00
Nikita Mikhaylov
8c06abee73
Merge pull request #25902 from Avogar/arrow-nested 
Refactor ArrowColumnToCHColumn, support inserting Nested as Array(Struct) in Arrow/ORC/Parquet
 2021-08-12 20:02:01 +03:00
mergify[bot]
80eaea1c51
Merge branch 'master' into short-circuit 2021-08-12 08:38:45 +00:00
Alexey Milovidov
c590a239d5 Merge branch 'master' into fix-bad-cast 2021-08-11 05:48:37 +03:00
Pavel Kruglov
b841a96c39 Refactor code 2021-08-10 14:31:15 +03:00
vdimir
dfccc85a33
Check const and lc nested can beome nullable in JoinCommon::convertColumnToNullable 2021-08-10 11:37:55 +03:00
Pavel Kruglov
70b51133c1 Try to simplify code 2021-08-09 18:01:08 +03:00
Pavel Kruglov
9fd4d14521 Fix style, build, add test 2021-08-09 17:57:50 +03:00
Pavel Kruglov
f16f660636 Fix PVS check 2021-08-09 17:57:50 +03:00
Pavel Kruglov
29440011ae Fix 2021-08-09 17:57:50 +03:00
Pavel Kruglov
a9cf67ecf2 Fix bugs, add tests 2021-08-09 17:57:49 +03:00
Pavel Kruglov
e701903ba6 Fix tests 2021-08-09 17:57:49 +03:00
Pavel Kruglov
76b3ae4301 Fix tests 2021-08-09 17:57:49 +03:00
Pavel Kruglov
a95d45a157 Some clean up 2021-08-09 17:57:02 +03:00
Pavel Kruglov
170f98572a Update IColumn.h 2021-08-09 17:54:49 +03:00
Pavel Kruglov
e5460c2b3f Fix ColumnFunction create 2021-08-09 17:54:49 +03:00
Pavel Kruglov
ab65b7f25d Minor changes 2021-08-09 17:54:49 +03:00
Pavel Kruglov
3ec2ab6156 Avoid using expand, fix bugs, add tests and comments 2021-08-09 17:54:49 +03:00
Pavel Kruglov
62d4f4b25c Small refactoring and adding comments 2021-08-09 17:54:14 +03:00
Pavel Kruglov
d8f590febe Remove includes 2021-08-09 17:54:14 +03:00
Pavel Kruglov
7ff11aff2d Some fixes, more tests 2021-08-09 17:54:14 +03:00