Commit Graph

631 Commits

Author SHA1 Message Date
Maksim Kita
af5ad2f370 Remove DecimalPaddedPODArray 2022-01-27 10:07:53 +00:00
Anton Popov
90b74c2aeb better unflattenTuple 2022-01-27 03:24:34 +03:00
Anton Popov
e8ce091e68 Merge remote-tracking branch 'upstream/master' into HEAD 2022-01-21 20:11:18 +03:00
Anton Popov
751d9e6a44 support nested in json type (wip) 2022-01-21 17:58:21 +03:00
Maksim Kita
6b9a5ab15e TypeId name fix 2022-01-20 13:51:00 +00:00
Alexey Milovidov
56b25f4543 Review fixes 2022-01-17 04:14:16 +03:00
Alexey Milovidov
7848ea7d60 Allow insertFrom for ColumnFunction 2022-01-16 06:18:35 +03:00
Anton Popov
aa515b7126 support nested in json type (wip) 2021-12-23 15:28:40 +03:00
Raúl Marín
88b8fd8b60 Merge remote-tracking branch 'blessed/master' into libcxx13_take2 2021-12-23 09:16:19 +01:00
Dmitry Novik
38c1e81752 fixup 2021-12-21 22:00:08 +03:00
Dmitry Novik
56917ca0ff Fix WriteBuffer initialization 2021-12-21 18:49:20 +03:00
mergify[bot]
7da6d8b310
Merge branch 'master' into fix_IP_conversion 2021-12-21 12:16:05 +00:00
Raúl Marín
36164a72c2 Missing include 2021-12-20 18:17:10 +01:00
Maksim Kita
20e4ce3314 Short circuit evaluation function throwIf support 2021-12-20 14:49:05 +03:00
mergify[bot]
4580af9f79
Merge branch 'master' into fix_segfault_object_used_after_move 2021-12-19 04:33:03 +00:00
N. Kolotov
b059154a81 Fixed segfault when smart pointer is dereferenced after move. 2021-12-18 23:13:42 +03:00
Anton Popov
99ebabd822 Merge remote-tracking branch 'upstream/master' into HEAD 2021-12-17 19:02:29 +03:00
Anton Popov
a70e5c1b2d support nested in json type (wip) 2021-12-17 18:08:34 +03:00
Anton Popov
079b08a28a support nested in json type (wip) 2021-12-16 14:42:12 +03:00
mergify[bot]
4300f77532
Merge branch 'master' into fix_IP_conversion 2021-12-14 20:12:34 +00:00
Anton Popov
ea67abf3f0 fix custom serializations in native protocol 2021-12-08 21:59:36 +03:00
Anton Popov
d8367334a3 Merge remote-tracking branch 'upstream/master' into HEAD 2021-12-08 18:26:19 +03:00
Azat Khuzhin
2b1789adc5 Pass through version for aggregate function from SerializationAggregateFunction too
Suggested-by: @kssenii
2021-12-05 21:45:07 +03:00
Azat Khuzhin
428e34d811 Fix versioning of aggregate functions (fixes performance tests) 2021-12-05 21:45:07 +03:00
Anton Popov
6f4d9a53b2 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-12-01 15:54:33 +03:00
Anton Popov
54f51444c0 Merge remote-tracking branch 'upstream/master' into HEAD 2021-12-01 15:49:02 +03:00
Anton Popov
63fceca6a8 support nested in json type (wip) 2021-12-01 05:59:56 +03:00
kssenii
be3b4ca8fe Merge branch 'master' of github.com:ClickHouse/ClickHouse into versioning 2021-11-27 09:44:31 +00:00
alexey-milovidov
423d497b27
Merge pull request #31742 from CurtizJ/fix-map
Fix usage of `Buffer` table engine with type `Map`
2021-11-25 04:56:37 +03:00
Anton Popov
37350e007c fix mutate of ColumnMap 2021-11-25 00:05:08 +03:00
Anton Popov
ccd78e3838 Merge remote-tracking branch 'upstream/master' into HEAD 2021-11-22 17:19:35 +03:00
mergify[bot]
a7ba3e23a0
Merge branch 'master' into fix-write-buffers 2021-11-22 11:24:27 +00:00
Alexey Milovidov
e898c6608c Fix broken non-x86 builds 2021-11-20 15:58:44 +03:00
kssenii
37f482d478 Merge branch 'master' of github.com:ClickHouse/ClickHouse into versioning 2021-11-15 07:31:11 +00:00
avogar
c521a9131a Small refactoring of WriteBiffer-s 2021-11-11 02:11:18 +03:00
Anton Popov
a20922b2d3 Merge remote-tracking branch 'origin/sparse-serialization' into HEAD 2021-11-09 15:36:25 +03:00
mergify[bot]
4f6fd49266
Merge branch 'master' into fix_IP_conversion 2021-11-09 12:11:54 +00:00
Anton Popov
a7f219b3aa better sampling in ColumnSparse and fix alter rename column 2021-11-08 21:25:31 +03:00
Anton Popov
84e914e05a minor fixes near serializations 2021-11-05 01:46:00 +03:00
Anton Popov
46fa062a81 fix tests 2021-11-02 23:30:28 +03:00
Anton Popov
d50137013c Merge remote-tracking branch 'upstream/master' into HEAD 2021-11-01 16:55:53 +03:00
Anton Popov
0099dfd523 refactoring of SerializationInfo 2021-10-29 20:21:02 +03:00
alexey-milovidov
69bc2a8d2a
Merge pull request #30780 from ClickHouse/remove-cruft
Remove cruft
2021-10-28 17:42:18 +03:00
Alexey Milovidov
8b4a6a2416 Remove cruft 2021-10-28 02:10:39 +03:00
jasperzhu
49dd69c6c9 adjust header files 2021-10-27 05:22:05 +05:30
jasperzhu
0217ed09ac revise compile flag 2021-10-27 05:19:15 +05:30
jasperzhu
5ab55f7eaa fixed function defect 2021-10-27 04:21:46 +05:30
jasperzhu
27f30dab89 remove function due to extra latency 2021-10-27 03:44:28 +05:30
jasperzhu
fb31410926 change to inline for bitmask generator 2021-10-27 00:05:47 +05:30
jasperzhu
1caff8a81c fixed compile error 2021-10-26 22:12:21 +05:30
jasperzhu
911b9c3973 Improve function defect 2021-10-26 06:02:57 +05:30
jasperzhu
3d115536f5 Improve filter bitmask generator function all in one with sse/avx2/avx512 2021-10-26 01:39:09 +05:30
Maksim Kita
243be5028a
Merge pull request #30014 from jinjunzh/master
add x86 feature avx2/avx512 support for accelerating filter operation
2021-10-22 14:16:47 +03:00
vesslanjin
f607a5d0ab remove branch in ColumnDecimal, same as pull req #29881
Co-authored-by: Zhu Jasper jasper.zhu@intel.com

Signed-off-by: vesslanjin <jun.i.jin@intel.com>
2021-10-20 07:40:06 -04:00
Anton Popov
d71ffc355a Merge remote-tracking branch 'upstream/master' into HEAD 2021-10-18 15:18:22 +03:00
Anton Popov
e46396331d better permute in ColumnSparse 2021-10-14 20:28:31 +03:00
Anton Popov
7aa6068fb2 Merge remote-tracking branch 'upstream/master' into HEAD 2021-10-14 19:44:08 +03:00
jasperzhu
f08dd0e8d8 modify flags definitions 2021-10-13 05:13:34 +05:30
jasperzhu
a1ab3b6606 Fixed code style issue 2021-10-12 22:44:37 +05:30
jasperzhu
8f349907fb Fixed trailing whitespaces issue 2021-10-12 09:21:54 +05:30
jasperzhu
3be83a26d0 fixed Typos and Styles issues 2021-10-12 08:24:47 +05:30
jasperzhu
0ef2624408 add specific flags for compiling filter operation source files 2021-10-12 04:04:27 +05:30
jasperzhu
a3d629a5b5 add x86 feature avx2/avx512 support for filter implementation 2021-10-11 22:51:13 +05:30
Nikolai Kochetov
a95c28ec4b
Merge pull request #29898 from ClickHouse/remove-native-stream
Remove some more streams.
2021-10-10 21:01:16 +03:00
Vladimir C
3940382c2f
Remove defined(__POPCNT__) from column filter 2021-10-09 13:34:02 +03:00
vesslanjin
bed09ee68d Whitespace fix
Signed-off-by: vesslanjin <jun.i.jin@intel.com>
2021-10-09 02:56:10 -04:00
vesslanjin
75af011068 use while instead of popcnt + for
Co-authored-by: Zhu Jasper jasper.zhu@intel.com
2021-10-08 14:58:23 -04:00
Nikolai Kochetov
340b53ef85 Remove some more streams. 2021-10-08 17:03:54 +03:00
Vladimir C
525df99893
Whitespace fix 2021-10-08 12:10:19 +03:00
Vladimir C
35feb08236
Whitespace changes 2021-10-08 12:03:55 +03:00
vesslanjin
383f3a3a20 Remove branchy code in filter operation with a better implementation with popcnt/ctz which have better performance
Co-authored-by: Zhu Jasper <jasper.zhu@intel.com>
2021-10-08 09:19:58 -04:00
Nikolai Kochetov
25e2ebac75
Merge pull request #29782 from azat/lc-concurrent-GROUP-BY-fix
Fix concurrent access to LowCardinality during GROUP BY (leads to SIGSEGV)
2021-10-07 16:27:53 +03:00
Vladimir C
4b80c6ff21
Merge pull request #29545 from CurtizJ/remove-permute 2021-10-07 11:07:34 +03:00
mergify[bot]
5b667d3cd5
Merge branch 'master' into fix_IP_conversion 2021-10-06 16:40:47 +00:00
Anton Popov
199ac31b48
Merge pull request #29683 from myrrc/improvement/tl-to-common
Non-recursive implementation for type list and its functions
2021-10-05 21:49:15 +03:00
Azat Khuzhin
342ec02664 Fix concurrent access to LowCardinality during GROUP BY (leads to SIGSEGV)
The problem is that GROUP BY can update saved_hash, which can be also
updated by subsequent update of a dictionary, and this will lead to
use-after-free.

You will find ASan report in `details`.

<details>

    ==24679==ERROR: AddressSanitizer: heap-use-after-free on address 0x604000615d20 at pc 0x000022cc8684 bp 0x7ffea6b5f850 sp 0x7ffea6b5f848
    READ of size 8 at 0x604000615d20 thread T223 (QueryPipelineEx)
        0 0x22cc8683 in DB::ReverseIndex<>::insert(StringRef const&) obj-x86_64-linux-gnu/../src/Columns/ReverseIndex.h
        1 0x22cc0de1 in COW<DB::IColumn>::mutable_ptr<DB::IColumn> DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeImpl<char8_t>()::'lambda'()::operator()() const obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:540:39
        2 0x22cc0de1 in COW<DB::IColumn>::mutable_ptr<DB::IColumn> DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeImpl<char8_t>() obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:570:23
        3 0x22cb9c66 in COW<DB::IColumn>::mutable_ptr<DB::IColumn> DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeFrom()::'lambda'(auto)::operator()<char8_t>(auto) const obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:592:26
        4 0x22cb9c66 in DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeFrom() obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:600:28
        5 0x2500b897 in DB::ColumnLowCardinality::insertRangeFrom() obj-x86_64-linux-gnu/../src/Columns/ColumnLowCardinality.cpp:205:62
        6 0x25a182f4 in DB::appendBlock(DB::Block const&, DB::Block&) obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:470:23
        7 0x25a182f4 in DB::BufferSink::insertIntoBuffer(DB::Block const&, DB::StorageBuffer::Buffer&) obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:634:9
        8 0x25a173cc in DB::BufferSink::consume(DB::Chunk) obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:595:9
        9 0x26d1c997 in DB::SinkToStorage::transform(DB::Chunk&) obj-x86_64-linux-gnu/../src/Processors/Sinks/SinkToStorage.cpp:18:5

    0x604000615d20 is located 16 bytes inside of 40-byte region [0x604000615d10,0x604000615d38)
    freed by thread T37 (QueryPipelineEx) here:
        2 0x22cb9392 in boost::intrusive_ptr<DB::ColumnVector<unsigned long> >::~intrusive_ptr() obj-x86_64-linux-gnu/../contrib/boost/boost/smart_ptr/intrusive_ptr.hpp:98:23
        4 0x22cb9392 in COW<DB::IColumn>::mutable_ptr<DB::ColumnVector<unsigned long> >::operator=() obj-x86_64-linux-gnu/../src/Common/COW.h💯57
        5 0x22cb9392 in DB::ReverseIndex<>::tryGetSavedHash() const obj-x86_64-linux-gnu/../src/Columns/ReverseIndex.h:362:28
        6 0x22cb9392 in DB::ColumnUnique<DB::ColumnString>::tryGetSavedHash() const obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:125:76
        7 0x242eaed3 in DB::ColumnsHashing::HashMethodSingleLowCardinalityColumn<>::HashMethodSingleLowCardinalityColumn() obj-x86_64-linux-gnu/../src/Common/ColumnsHashing.h:287:50
        8 0x242206c6 in void DB::Aggregator::executeImpl<>() const obj-x86_64-linux-gnu/../src/Interpreters/Aggregator.cpp:596:28
        9 0x24148e99 in DB::Aggregator::executeOnBlock() const obj-x86_64-linux-gnu/../src/Interpreters/Aggregator.cpp:1004:9
        10 0x26c24f3a in DB::AggregatingTransform::consume(DB::Chunk) obj-x86_64-linux-gnu/../src/Processors/Transforms/AggregatingTransform.cpp:539:33
        11 0x26c2054e in DB::AggregatingTransform::work() obj-x86_64-linux-gnu/../src/Processors/Transforms/AggregatingTransform.cpp:500:9

    previously allocated by thread T37 (QueryPipelineEx) here:
        0 0xb6d44fd in operator new(unsigned long) (/src/ch/tmp/upstream/clickhouse-asan+0xb6d44fd)
        1 0x11b78580 in COW<DB::IColumn>::mutable_ptr<> COWHelper<>::create<unsigned long&>(unsigned long&) (/src/ch/tmp/upstream/clickhouse-asan+0x11b78580)
        2 0x22cbf7b1 in DB::ReverseIndex<>::calcHashes() const obj-x86_64-linux-gnu/../src/Columns/ReverseIndex.h:472:17
        3 0x22cc2307 in DB::ReverseIndex<>::buildIndex() obj-x86_64-linux-gnu/../src/Columns/ReverseIndex.h:438:22
        4 0x22cc658c in DB::ReverseIndex<>::insert(StringRef const&) obj-x86_64-linux-gnu/../src/Columns/ReverseIndex.h:484:9
        5 0x22cc0de1 in COW<DB::IColumn>::mutable_ptr<DB::IColumn> DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeImpl<char8_t>()::'lambda'()::operator()() const obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:540:39
        6 0x22cc0de1 in COW<DB::IColumn>::mutable_ptr<DB::IColumn> DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeImpl<char8_t>() obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:570:23
        7 0x22cb9c66 in COW<DB::IColumn>::mutable_ptr<DB::IColumn> DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeFrom()::'lambda'(auto)::operator()<char8_t>(auto) const obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:592:26
        8 0x22cb9c66 in DB::ColumnUnique<DB::ColumnString>::uniqueInsertRangeFrom() obj-x86_64-linux-gnu/../src/Columns/ColumnUnique.h:600:28
        9 0x2500b897 in DB::ColumnLowCardinality::insertRangeFrom() obj-x86_64-linux-gnu/../src/Columns/ColumnLowCardinality.cpp:205:62
        10 0x25a182f4 in DB::appendBlock(DB::Block const&, DB::Block&) obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:470:23
        11 0x25a182f4 in DB::BufferSink::insertIntoBuffer(DB::Block const&, DB::StorageBuffer::Buffer&) obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:634:9
        12 0x25a173cc in DB::BufferSink::consume(DB::Chunk) obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:595:9
        13 0x26d1c997 in DB::SinkToStorage::transform(DB::Chunk&) obj-x86_64-linux-gnu/../src/Processors/Sinks/SinkToStorage.cpp:18:5

    SUMMARY: AddressSanitizer: heap-use-after-free obj-x86_64-linux-gnu/../src/Columns/ReverseIndex.h in DB::ReverseIndex<unsigned long, DB::ColumnString>::insert(StringRef const&)
    Shadow bytes around the buggy address:
      0x0c08800bab50: fa fa fd fd fd fd fd fa fa fa 00 00 00 00 00 00
      0x0c08800bab60: fa fa fd fd fd fd fd fa fa fa 00 00 00 00 00 fa
      0x0c08800bab70: fa fa 00 00 00 00 00 00 fa fa 00 00 00 00 00 00
      0x0c08800bab80: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fa
      0x0c08800bab90: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
    =>0x0c08800baba0: fa fa fd fd[fd]fd fd fa fa fa fd fd fd fd fd fa
      0x0c08800babb0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fa
      0x0c08800babc0: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
      0x0c08800babd0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fa
      0x0c08800babe0: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
      0x0c08800babf0: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fd
    Shadow byte legend (one shadow byte represents 8 application bytes):
      Addressable:           00
      Partially addressable: 01 02 03 04 05 06 07
      Heap left redzone:       fa
      Freed heap region:       fd
      Stack left redzone:      f1
      Stack mid redzone:       f2
      Stack right redzone:     f3
      Stack after return:      f5
      Stack use after scope:   f8
      Global redzone:          f9
      Global init order:       f6
      Poisoned by user:        f7
      Container overflow:      fc
      Array cookie:            ac
      Intra object redzone:    bb
      ASan internal:           fe
      Left alloca redzone:     ca
      Right alloca redzone:    cb
    ==24679==ABORTING

</details>
2021-10-05 21:07:10 +03:00
Anton Popov
b5098301d6 fix build 2021-10-05 18:23:53 +03:00
Anton Popov
cd08057285 better arguments 2021-10-04 15:02:11 +03:00
Mike Kot
8c8bada56e Non-recursive implementation for TL and its functions 2021-10-03 18:02:04 +02:00
Anton Popov
857b8f0b1d fix clang-tidy 2021-10-02 22:36:30 +03:00
Anton Popov
1f10246a10 Merge remote-tracking branch 'upstream/master' into HEAD 2021-10-02 22:33:27 +03:00
Alexey Milovidov
fe6b7c77c7 Rename "common" to "base" 2021-10-02 10:13:14 +03:00
Anton Popov
9c296e94f3 fix performance of updatePermutation 2021-10-01 19:45:17 +03:00
Anton Popov
515cc74530 generalize code in IColumn::updatePermutation 2021-10-01 00:11:54 +03:00
Anton Popov
0dfe96fc02 generalize code in IColumn::permute 2021-09-29 20:56:10 +03:00
Dmitry Novik
abdd8fb516 Cleanup 2021-09-29 19:42:41 +03:00
Anton Popov
914781052e Merge remote-tracking branch 'upstream/master' into HEAD 2021-09-29 17:37:07 +03:00
Anton Popov
dc94d2239e better ColumnSparse 2021-09-29 17:18:41 +03:00
Dmitry Novik
95193e06f6 Remove empty row creation 2021-09-28 16:59:58 +03:00
mergify[bot]
86eb589c72
Merge branch 'master' into fix_IP_conversion 2021-09-27 14:41:56 +00:00
alexey-milovidov
e4363872ca
Merge branch 'master' into yandex-to-clickhouse-in-configs 2021-09-25 19:58:36 +03:00
Alexey Milovidov
cd7f9d981c Remove ya.make 2021-09-25 04:22:54 +03:00
Alexey Milovidov
e9e77b4403 .tech -> .com 2021-09-22 03:22:57 +03:00
Dmitry Novik
9049fd701f Refactor ColumnStringHelpers 2021-09-21 17:27:34 +03:00
Vasily Nemkov
175b94fa2d Special case of conversion code for types with custom serialization/deserialization
Fixes CASTing from String or FixedString to IPv4 or IPv6 and back.
2021-09-21 15:34:06 +03:00
Anton Popov
eef436fe22 Merge remote-tracking branch 'upstream/master' into HEAD 2021-09-16 18:07:42 +03:00
Anton Popov
348537faee better IColumn::createWithOffsets 2021-09-16 16:57:45 +03:00
Anton Popov
3adc2a9aea fix distinct with sparse columns 2021-09-16 15:52:59 +03:00
Anton Popov
b0c0b2bc10 fix some builds 2021-09-14 19:33:00 +03:00
Anton Popov
d09f973b83 dynamic columns: fix several cases of parsing json 2021-09-13 17:02:38 +03:00