thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-13 09:52:38 +00:00
Code Issues Packages Projects Releases Wiki Activity
99,804 Commits 9,324 Branches 1,783 Tags 2.1 GiB
85193ede42
Commit Graph

3513 Commits

Author SHA1 Message Date
Alexey Milovidov
c09604bef2
Merge pull request #41363 from kitaisreal/small-style-fixes 
Small style fixes
 2022-09-17 00:19:00 +03:00
Nikita Taranov
6f186d3dd2
Do not return empty blocks from ConvertingAggregatedToChunksTransform (#41152) 
* impl

* add test

* update test
 2022-09-16 21:54:36 +02:00
avogar
0101cc2e56 Support complex combinators in window transform, arrayReduce*, initializeAggregation and Aggregate functons versionning 2022-09-16 19:07:36 +00:00
Igor Nikonov
24de5530d6 Remove unnecessary headers 2022-09-16 17:16:32 +00:00
Igor Nikonov
b34ec6453b Fix review comments 
+ one more test
 2022-09-16 17:11:08 +00:00
Kruglov Pavel
2d4a6b38af
Merge branch 'master' into fix-totals-extremes 2022-09-16 15:03:59 +02:00
Igor Nikonov
eeecaf7a31 Merge remote-tracking branch 'origin/master' into distinct_in_order_wo_order_by 2022-09-16 10:30:52 +00:00
Igor Nikonov
ba1f7de243 Fix review comments 2022-09-16 10:30:30 +00:00
Alexey Milovidov
da01982652
Merge pull request #41046 from azat/build/llvm-15 
Switch to llvm/clang 15
 2022-09-16 07:31:06 +03:00
Maksim Kita
276d3628e9 Small style fixes 2022-09-15 13:05:18 +02:00
Igor Nikonov
da75d28019 Merge remote-tracking branch 'origin/master' into distinct_in_order_wo_order_by 2022-09-15 10:54:42 +00:00
Kruglov Pavel
73cf72a5a4
Merge pull request #41309 from Avogar/fix-msgpack 
Add column type check before UUID insertion in MsgPack format
 2022-09-15 11:37:57 +02:00
Igor Nikonov
f011f72114 Try to apply optimization only if optimizer reaches pre distinct node 
+ more comments
 2022-09-14 20:55:49 +00:00
Igor Nikonov
bab5744a1d Merge remote-tracking branch 'origin/master' into distinct_in_order_wo_order_by 2022-09-14 17:54:46 +00:00
Igor Nikonov
d91ea0c5ab Fix + tests 2022-09-14 17:54:05 +00:00
avogar
59e7eb084c Add column type check before UUID insertion in MsgPack format 2022-09-14 11:15:10 +00:00
Kruglov Pavel
3396ff6c3a
Merge pull request #40516 from zjial/record_errors_for_import_by_csv 
Record errors while reading text formats (CSV, TSV).
 2022-09-14 12:52:32 +02:00
Igor Nikonov
c1dc77cd50 Merge remote-tracking branch 'origin/master' into distinct_in_order_wo_order_by 2022-09-13 18:24:48 +00:00
Igor Nikonov
1be5a79e9f Exclude const columns in distinct when matching with sorting key 
+ tests
 2022-09-13 18:23:55 +00:00
Igor Nikonov
c04d46bc96 Update data stream properties after reading in order applied 
+ tests
 2022-09-13 17:50:43 +00:00
Kruglov Pavel
110be0688e
Merge pull request #40909 from ClickHouse/Avogar-patch-1 
Make better exception message in schema inference
 2022-09-13 14:44:29 +02:00
Kruglov Pavel
17621b5607
Merge branch 'master' into fix-totals-extremes 2022-09-13 14:36:31 +02:00
Igor Nikonov
c129b31d7b Test fix 2022-09-13 09:40:23 +00:00
zhenjial
5841d9e9b0 sync before destruct 2022-09-13 15:53:24 +08:00
zhenjial
67c08e3e22 sync before destruct 2022-09-13 15:06:22 +08:00
zhenjial
16c8cd0bd3 wait write finish 2022-09-13 14:19:40 +08:00
zhongyuankai
1477bb132f
Merge branch 'master' into compress_marks_and_primary_key 2022-09-13 11:05:57 +08:00
Igor Nikonov
1c9c303cce Fix typos 2022-09-12 21:11:05 +00:00
Igor Nikonov
7502be3b75 Fixes 2022-09-12 20:11:24 +00:00
Kruglov Pavel
702ddff5f6
Fix style 2022-09-12 19:38:34 +02:00
Kruglov Pavel
060adfbe93
Merge branch 'master' into new-json-formats 2022-09-12 19:37:46 +02:00
avogar
8ac2fc7b26 Don't outout totals/extremes in all row formats, update docs 2022-09-12 17:21:40 +00:00
Igor Nikonov
bdd7c7c2c1 Merge remote-tracking branch 'origin/master' into distinct_in_order_wo_order_by 2022-09-12 17:02:23 +00:00
Igor Nikonov
9fc0aaf477 Something working with tests 2022-09-12 17:01:26 +00:00
Alexey Milovidov
2aedd41023
Remove strange code (#40195) 
* Remove strange code

* Even more code removal

* Fix style

* Remove even more code

* Simplify code by making it slower

* Attempt to do something

* Attempt to do something

* Well do something with this horrible trash

* Add a test
 2022-09-12 16:29:23 +02:00
Kruglov Pavel
6535301888
Merge branch 'master' into Avogar-patch-1 2022-09-12 12:23:28 +02:00
Alexey Milovidov
7f1e7b5967 Merge branch 'master' into fix-half-of-trash 2022-09-11 06:20:47 +02:00
Azat Khuzhin
c1e70169d2 Suppress clang-analyzer-cplusplus.NewDelete in MsgPackRowInputFormat 
Appartently there is some issue with clang-15, since even the following
example shows error [1].

  [1]: https://gist.github.com/azat/027f0e949ea836fc2e6269113ceb8752

clang-tidy report [1]:

    FAILED: src/CMakeFiles/dbms.dir/Processors/Formats/Impl/MsgPackRowInputFormat.cpp.o                                                                                                            /usr/bin/cmake -E __run_co_compile --launcher="prlimit;--as=10000000000;--data=5000000000;--cpu=1000;/usr/bin/ccache" --tidy=/usr/bin/clang-tidy-15 --source=/ch/src/Processors/Formats/Impl/MsgPackRowInputFormat.cpp -- /usr/bin/clang++-15 --target=x86_64-linux-gnu --sysroot=/ch/cmake/linux/../../contrib/sysroot/linux-x86_64/x86_64-linux-gnu/libc  -DAWS_SDK_VERSION_MAJOR=1 -DAWS_SDK_VERSION_MINOR=7 -DAWS_SDK_VERSION_PATCH=231 -DBOOST_ASIO_HAS_STD_INVOKE_RESULT=1 -DBOOST_ASIO_STANDALONE=1 -DCARES_STATICLIB -DCONFIGDIR=\"\" -DENABLE_MULTITARGET_CODE=1 -DENABLE_OPENSSL_ENCRYPTION -DHAS_RESERVED_IDENTIFIER -DHAVE_CONFIG_H -DLIBSASL_EXPORTS=1 -DLZ4_DISABLE_DEPRECATE_WARNINGS=1 -DOBSOLETE_CRAM_ATTR=1 -DOBSOLETE_DIGEST_ATTR=1 -DPLUGINDIR=\"\" -DPOCO_ENABLE_CPP11 -DPOCO_HAVE_FD_EPOLL -DPOCO_OS_FAMILY_UNIX -DSASLAUTHD_CONF_FILE_DEFAULT=\"\" -DSNAPPY_CODEC_AVAILABLE -DSTD_EXCEPTION_HAS_STACK_TRACE=1 -DUNALIGNED_OK -DWITH_COVERAGE=0 -DWITH_GZFILEOP -DX86_64 -DZLIB_COMPAT -D_LIBCPP_ENABLE_THREAD_SAFETY_ANNOTATIONS -Iincludes/configs -I/ch/src -Isrc -Isrc/Core/include -I/ch/base/glibc-compatibility/memcpy -I/ch/base/base/.. -Ibase/base/.. -I/ch/contrib/cctz/include -I/ch/base/pcg-random/. -I/ch/contrib/miniselect/include -I/ch/contrib/zstd/lib -Icontrib/cyrus-sasl-cmake -I/ch/contrib/lz4/lib -I/ch/src/Common/mysqlxx/. -Icontrib/c-ares -I/ch/contrib/c-ares -I/ch/contrib/c-ares/include -isystem /ch/contrib/libcxx/include -isystem /ch/contrib/libcxxabi/include -isystem /ch/contrib/libunwind/include -isystem /ch/contrib/libdivide/. -isystem /ch/contrib/jemalloc-cmake/include -isystem /ch/contrib/llvm/llvm/include -isystem contrib/llvm/llvm/include -isystem /ch/contrib/abseil-cpp -isystem /ch/contrib/croaring/cpp -isystem /ch/contrib/croaring/include -isystem /ch/contrib/cityhash102/include -isystem /ch/contrib/boost -isystem /ch/contrib/poco/Net/include -isystem /ch/contrib/poco/Foundation/include -isystem /ch/contrib/poco/NetSSL_OpenSSL/include -isystem /ch/contrib/poco/Crypto/include -isystem /ch/contrib/boringssl/include -isystem /ch/contrib/poco/Util/include -isystem /ch/contrib/poco/JSON/include -isystem /ch/contrib/poco/XML/include -isystem /ch/contrib/replxx/include -isystem /ch/contrib/fmtlib-cmake/../fmtlib/include -isystem /ch/contrib/magic_enum/include -isystem /ch/contrib/double-conversion -isystem /ch/contrib/dragonbox/include -isystem /ch/contrib/re2 -isystem contrib/re2-cmake -isystem /ch/contrib/zlib-ng -isystem contrib/zlib-ng-cmake -isystem /ch/contrib/pdqsort -isystem /ch/contrib/xz/src/liblzma/api -isystem /ch/contrib/aws-c-common/include -isystem /ch/contrib/aws-c-event-stream/include -isystem /ch/contrib/aws/aws-cpp-sdk-s3/include -isystem /ch/contrib/aws/aws-cpp-sdk-core/include -isystem contrib/aws-s3-cmake/include -isystem /ch/contrib/snappy -isystem contrib/snappy-cmake -isystem /ch/contrib/msgpack-c/include -isystem /ch/contrib/fast_float/include -isystem /ch/contrib/librdkafka-cmake/include -isystem /ch/contrib/librdkafka/src -isystem contrib/librdkafka-cmake/auxdir -isystem /ch/contrib/cppkafka/include -isystem /ch/contrib/nats-io/src -isystem /ch/contrib/nats-io/src/adapters -isystem /ch/contrib/nats-io/src/include -isystem /ch/contrib/nats-io/src/unix -isystem /ch/contrib/libuv/include -isystem /ch/contrib/krb5/src/include -isystem contrib/krb5-cmake/include -isystem /ch/contrib/NuRaft/include -isystem /ch/contrib/poco/MongoDB/include -isystem contrib/mariadb-connector-c-cmake/include-public -isystem /ch/contrib/mariadb-connector-c/include -isystem /ch/contrib/mariadb-connector-c/libmariadb -isystem /ch/contrib/icu/icu4c/source/i18n -isystem /ch/contrib/icu/icu4c/source/common -isystem /ch/contrib/capnproto/c++/src -isystem /ch/contrib/arrow/cpp/src -isystem /ch/contrib/arrow-cmake/cpp/src -isystem contrib/arrow-cmake/cpp/src -isystem contrib/arrow-cmake/../orc/c++/include -isystem /ch/contrib/orc/c++/include -isystem contrib/avro-cmake/include -isystem /ch/contrib/avro/lang/c++/api -isystem /ch/contrib/openldap-cmake/linux_x86_64/include -isystem /ch/contrib/openldap/include -isystem /ch/contrib/sparsehash-c11 -isystem /ch/contrib/protobuf/src -isystem src/Server/grpc_protos -isystem /ch/contrib/grpc/include -isystem /ch/contrib/libhdfs3/include -isystem /ch/contrib/hive-metastore -isystem /ch/contrib/thrift/lib/cpp/src -isystem contrib/thrift-cmake -isystem /ch/contrib/azure/sdk/core/azure-core/inc-isystem /ch/contrib/azure/sdk/identity/azure-identity/inc -isystem /ch/contrib/azure/sdk/storage/azure-storage-common/inc -isystem /ch/contrib/azure/sdk/storage/azure-storage-blobs/inc -isystem /ch/contrib/s2geometry/src -isystem /ch/contrib/AMQP-CPP/include -isystem /ch/contrib/AMQP-CPP -isystem /ch/contrib/sqlite-amalgamation -isystem /ch/contrib/rocksdb/include -isystem /ch/contrib/libpqxx/include -isystem /ch/contrib/libpq -isystem /ch/contrib/libpq/include -isystem /ch/contrib/libstemmer_c/include -isystem /ch/contrib/wordnet-blast -isystem /ch/contrib/lemmagen-c/include -isystem /ch/contrib/simdjson/include -isystem /ch/contrib/rapidjson/include -isystem /ch/contrib/consistent-hashing --gcc-toolchain=/ch/cmake/linux/../../contrib/sysroot/linux-x86_64 -std=c++20 -fdiagnostics-color=always -Xclang -fuse-ctor-homing -fsized-deallocation  -UNDEBUG -gdwarf-aranges -pipe -mssse3 -msse4.1 -msse4.2 -mpclmul -mpopcnt -fasynchronous-unwind-tables -falign-functions=32 -mbranches-within-32B-boundaries -fdiagnostics-absolute-paths -fstrict-vtable-pointers -fexperimental-new-pass-manager -Wall -Wextra -Weverything -Wpedantic -Wno-zero -length-array -Wno-c++98-compat-pedantic -Wno-c++98-compat -Wno-c++20-compat -Wno-conversion -Wno-ctad-maybe-unsupported -Wno-disabled-macro-expansion -Wno-documentation-unknown-command -Wno-double-promotion -Wno-exit-time-destructors -Wno-float-equal -Wno-global-constructors -Wno-missing-prototypes -Wno-missing-variable-declarations -Wno-padded -Wno-switch-enum -Wno-undefined-func-template -Wno-unused-template -Wno-vla -Wno-weak-template-vtables -Wno-weak-vtables -Wno-thread-safety-negative -g -O0 -g -gdwarf-4 -fno-inline  -D_LIBCPP_DEBUG=0   -D OS_LINUX -I/ch/base -I/ch/contrib/magic_enum/include -include /ch/src/Core/iostream_debug_helpers.h -Werror -nostdinc++ -std=gnu++2a -MD -MT src/CMakeFiles/dbms.dir/Processors/Formats/Impl/MsgPackRowInputFormat.cpp.o -MF src/CMakeFiles/dbms.dir/Processors/Formats/Impl/MsgPackRowInputFormat.cpp.o.d -o src/CMakeFiles/dbms.dir/Processors/Formats/Impl/MsgPackRowInputFormat.cpp.o -c /ch/src/Processors/Formats/Impl/MsgPackRowInputFormat.cpp

    /ch/contrib/msgpack-c/include/msgpack/v1/detail/cpp11_zone.hpp:195:9: error: Attempt to free released memory [clang-analyzer-cplusplus.NewDelete,-warnings-as-errors]
            ::free(p);
            ^
    /ch/src/Processors/Formats/Impl/MsgPackRowInputFormat.cpp:509:5: note: Taking false branch
        if (buf.eof())
        ^
    /ch/src/Processors/Formats/Impl/MsgPackRowInputFormat.cpp:514:24: note: Assuming 'i' is not equal to field 'number_of_columns'
        for (size_t i = 0; i != number_of_columns; ++i)
                           ^~~~~~~~~~~~~~~~~~~~~~
    /ch/src/Processors/Formats/Impl/MsgPackRowInputFormat.cpp:514:5: note: Loop condition is true.  Entering loop body
        for (size_t i = 0; i != number_of_columns; ++i)
        ^
    /ch/src/Processors/Formats/Impl/MsgPackRowInputFormat.cpp:516:30: note: Calling 'MsgPackSchemaReader::readObject'
            auto object_handle = readObject();
                                 ^~~~~~~~~~~~
    /ch/src/Processors/Formats/Impl/MsgPackRowInputFormat.cpp:426:5: note: Taking false branch
        if (buf.eof())
        ^
    /ch/src/Processors/Formats/Impl/MsgPackRowInputFormat.cpp:433:5: note: Loop condition is true.  Entering loop body
        while (need_more_data)
        ^
    /ch/src/Processors/Formats/Impl/MsgPackRowInputFormat.cpp:438:29: note: Calling 'unpack'
                object_handle = msgpack::unpack(buf.position(), buf.buffer().end() - buf.position(), offset);
                                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    /ch/contrib/msgpack-c/include/msgpack/v3/unpack.hpp:52:12: note: Calling 'unpack'
        return msgpack::v3::unpack(data, len, off, referenced, f, user_data, limit);
               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    /ch/contrib/msgpack-c/include/msgpack/v3/unpack.hpp:35:5: note: Control jumps to the 'default' case at line 40
        switch(ret) {
        ^
    /ch/contrib/msgpack-c/include/msgpack/v3/unpack.hpp:41:9: note:  Execution continues on line 43
            break;
            ^
    /ch/contrib/msgpack-c/include/msgpack/v3/unpack.hpp:43:35: note: Calling '~unique_ptr'
        return msgpack::object_handle();
                                      ^
    /ch/contrib/libcxx/include/__memory/unique_ptr.h:269:19: note: Calling 'unique_ptr::reset'
      ~unique_ptr() { reset(); }
                      ^~~~~~~
    /ch/contrib/libcxx/include/__memory/unique_ptr.h:314:9: note: '__tmp' is non-null
        if (__tmp)
            ^~~~~
    /ch/contrib/libcxx/include/__memory/unique_ptr.h:314:5: note: Taking true branch
        if (__tmp)
        ^
    /ch/contrib/libcxx/include/__memory/unique_ptr.h:315:7: note: Calling 'default_delete::operator()'
          __ptr_.second()(__tmp);
          ^~~~~~~~~~~~~~~~~~~~~~
    /ch/contrib/libcxx/include/__memory/unique_ptr.h:54:5: note: Memory is released
        delete __ptr;
        ^~~~~~~~~~~~
    /ch/contrib/libcxx/include/__memory/unique_ptr.h:54:5: note: Calling 'zone::operator delete'
        delete __ptr;
        ^~~~~~~~~~~~
    /ch/contrib/msgpack-c/include/msgpack/v1/detail/cpp11_zone.hpp:195:9: note: Attempt to free released memory
            ::free(p);
            ^~~~~~~~~

  [1]: https://s3.amazonaws.com/clickhouse-builds/41046/9677898b3b234a5ba0371edaf719ea8890d084ff/binary_tidy/build_log.log

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-09-10 21:38:35 +02:00
Alexey Milovidov
fd235919aa Remove some methods 2022-09-10 05:04:40 +02:00
Alexey Milovidov
fa62c7e982 Fix half of trash 2022-09-10 04:08:16 +02:00
Igor Nikonov
726639484a
Revert "Query plan optimization setting: read in window order" 2022-09-09 22:15:38 +02:00
Kruglov Pavel
c33aa54032
Fix 2022-09-09 17:53:26 +02:00
Kruglov Pavel
f669d305b6
Fix comment 2022-09-09 17:45:47 +02:00
zhenjial
bd9fabc3f7 code optimization, add test 2022-09-09 23:27:42 +08:00
avogar
ad68b7be0f Better 2022-09-09 15:01:45 +00:00
Igor Nikonov
5379f37331
Merge pull request #41015 from ClickHouse/window_func_optimization_setting 
Query plan optimization setting: read in window order
 2022-09-09 14:00:41 +02:00
avogar
46a0318a36 Support JSONColumnsWithMetadata input format 2022-09-08 17:58:44 +00:00
zhenjial
469ceaa156 code optimization 2022-09-09 00:47:43 +08:00
avogar
c380decbbb Make better, add new settings 2022-09-08 16:07:20 +00:00
avogar
545be27f81 Merge branch 'master' of github.com:ClickHouse/ClickHouse into new-json-formats 2022-09-08 13:48:10 +00:00
Kruglov Pavel
96cb0e54d8
Merge branch 'master' into Avogar-patch-1 2022-09-08 13:24:08 +02:00
Igor Nikonov
c23412ae77 slight changes 2022-09-07 22:02:50 +00:00
Kseniia Sumarokova
a270eeef91
Merge pull request #41008 from kssenii/refactor-merge-tree-read 
Small refactoring around merge tree readers (get rid of data part ptr)
 2022-09-07 18:27:33 +02:00
zhenjial
0f788d98f5 new implementation 2022-09-06 20:39:54 +08:00
zhenjial
18db90dcfc Record errors while reading text formats (CSV, TSV). 2022-09-06 17:19:15 +08:00
Duc Canh Le
6950016b8a fix grouping set with group_by_use_nulls 2022-09-06 09:39:27 +08:00
zhongyuankai
0bf76fe642 Merge branch 'compress-marks' into compress_marks_and_primary_key 2022-09-06 08:01:43 +08:00
Igor Nikonov
7a8b8e7a39 Optimizer setting: read in window order 
optimization's setting is checked before applying it, not inside the optimization code
 2022-09-05 20:47:11 +00:00
Igor Nikonov
30860290de Continue 2022-09-05 20:12:00 +00:00
kssenii
83514fa2ef Refactor 2022-09-05 20:08:22 +02:00
Alexey Milovidov
d7127e4b2d Make it slightly more sane 2022-09-05 07:26:58 +02:00
Igor Nikonov
8fece1e2d2
Merge branch 'master' into sort_mode_rename 2022-09-04 21:44:33 +02:00
Alexey Milovidov
193cd1b3b2
Merge pull request #39138 from nickitat/control_block_size_in_aggregator 
Control block size in aggregator
 2022-09-04 04:51:00 +03:00
Igor Nikonov
70f779b81d Just to save 2022-09-02 21:24:33 +00:00
Igor Nikonov
5d7fa55f36
Merge branch 'master' into sort_mode_rename 2022-09-02 23:19:04 +02:00
Kruglov Pavel
77071381e4
fix build 2022-09-02 16:37:33 +02:00
Vladimir C
963c0111bf
Merge pull request #39418 from vdimir/join_and_sets 
Filter joined streams for `full_sorting_join` by each other before sorting
 2022-09-02 13:57:06 +02:00
Antonio Andelic
e64436fef3 Fix typos with new codespell 2022-09-02 08:54:48 +00:00
Robert Schulze
319d8b00a7
Merge pull request #39010 from FrankChen021/tracing_context_propagation 
Improve the opentelemetry tracing context propagation across threads
 2022-09-02 07:56:43 +02:00
Robert Schulze
c7c00f9002
Merge pull request #40739 from ClickHouse/clang-tidy-for-headers 
Enable clang-tidy for headers
 2022-09-02 07:54:50 +02:00
avogar
afc34dca41 Add new JSON formats, add improvements and refactoring 2022-09-01 19:00:24 +00:00
Kruglov Pavel
7a4a65bc36
Make better exception message in schema inference 2022-09-01 20:36:08 +02:00
Kruglov Pavel
f53aa86a20
Merge pull request #40485 from arthurpassos/fix-parquet-chunked-array-deserialization 
Add support for extended (chunked) arrays for Parquet format
 2022-09-01 19:40:40 +02:00
Dmitry Novik
ddadb362cf
Merge pull request #39762 from quickhouse/betterorderbyoptimization 
Fixed `Unknown identifier (aggregate-function)` exception which appears when a user tries to calculate WINDOW ORDER BY/PARTITION BY expressions over aggregate functions
 2022-09-01 18:08:06 +02:00
Frank Chen
9d63cbe811 Merge 'origin/master' into tracing_context_propagation to resolve conflicts 2022-09-01 23:18:59 +08:00
Vladimir C
12e6fc4182
Merge branch 'master' into join_and_sets 2022-09-01 14:56:14 +02:00
Kseniia Sumarokova
c6c67a248d
Merge pull request #40792 from canhld94/ch_canh_intersect_distinct 
Implement intersect + except distinct
 2022-09-01 14:35:26 +02:00
Anton Popov
f7bdf07adc
Merge pull request #38715 from CurtizJ/fix-read-in-order-fixed-prefix 
Better support of `optimize_read_in_order` in case of fixed prefix of sorting key
 2022-09-01 12:59:18 +02:00
Robert Schulze
de64c6b103
Merge branch 'master' into clang-tidy-for-headers 2022-09-01 10:24:56 +02:00
Kruglov Pavel
86516d3bb4
Merge pull request #40740 from amosbird/row-policy-index-fix-1 
Use index when row_policy_filter is always false
 2022-08-31 18:46:14 +02:00
Robert Schulze
cedf75ed5e
Enable clang-tidy for headers 
clang-tidy now also checks code in header files. Because the analyzer
finds tons of issues, activate the check only for directory "base/" (see
file ".clang-tidy"). All other directories, in particular "src/" are
left to future work.

While many findings were fixed, some were not (and suppressed instead).
Reasons for this include: a) the file is 1:1 copypaste of a 3rd-party
lib (e.g. pcg_extras.h) and fixing stuff would make upgrades/fixes more
difficult b) a fix would have broken lots of using code
 2022-08-31 10:48:15 +00:00
Anton Popov
3504781529
Merge branch 'master' into fix-read-in-order-fixed-prefix 2022-08-30 23:32:43 +02:00
Dmitry Novik
0a8378d9cd
Merge branch 'master' into betterorderbyoptimization 2022-08-30 14:23:22 +02:00
vdimir
0f6f3c73b0
Minor fix 2022-08-30 11:57:28 +00:00
Duc Canh Le
8590cc46c4 implement intersect + except distinct 2022-08-30 18:09:01 +08:00
Frank Chen
f17d56b528 Merge branch 'master' into tracing_context_propagation 2022-08-30 14:24:36 +08:00
vdimir
24f62e8486
Throw an error in CreatingSetsOnTheFlyTransform in case of input for finished 2022-08-29 11:27:08 +00:00
vdimir
b0e2616aa9
Style fixes in CreateSetAndFilterOnTheFlyTransform and related 2022-08-29 11:26:21 +00:00
Anton Popov
2a3e012931
Merge branch 'master' into fix-read-in-order-fixed-prefix 2022-08-29 13:17:26 +02:00
vdimir
7915b6948f
Fix build after rebase 2022-08-29 09:49:16 +00:00
vdimir
afb6b7d9cf
Test plan and pipeline for filtering step for join 2022-08-29 09:49:15 +00:00
vdimir
afeff512b5
Aux port for ReadHeadBalancedProcessor is empty Block 2022-08-29 09:49:14 +00:00
vdimir
95f87dc34e
fix sanitizer assert in CreateSetAndFilterOnTheFlyStep 2022-08-29 09:49:12 +00:00
vdimir
c67ab33d90
small fix CreateSetAndFilterOnTheFlyStep 2022-08-29 09:49:11 +00:00
vdimir
51e02d09f6
set preserves_sorting = true for CreateSetAndFilterOnTheFlyStep 2022-08-29 09:49:10 +00:00
vdimir
714c53ab24
fix typos 2022-08-29 09:49:09 +00:00
vdimir
8e1632f824
Create sets for joins: better code 2022-08-29 09:49:08 +00:00
vdimir
7228091ff1
rename CreateSetAndFilterOnTheFlyTransform 2022-08-29 09:49:07 +00:00
vdimir
67a9acc8db
rename CreatingSetOnTheFlyStep -> CreateSetAndFilterOnTheFlyStep 2022-08-29 09:49:07 +00:00
vdimir
d82a75ae75
cleanup PingPongProcessor 2022-08-29 09:49:06 +00:00
vdimir
e472e13c70
move PingPongProcessor/ReadHeadBalancedProceesor into separate file 2022-08-29 09:49:05 +00:00
vdimir
51a51694d6
Create sets for joins: better code 2022-08-29 09:49:01 +00:00
vdimir
c778bba13f
Create sets for joins: wip 2022-08-29 09:47:00 +00:00
vdimir
31a167848d
Fix set finish condition in CreatingSetsOnTheFlyTransform 2022-08-29 09:46:59 +00:00
vdimir
71708d595f
Create sets for joins: wip 2022-08-29 09:46:59 +00:00
vdimir
8f06430ebd
Create sets for joins: upd 2022-08-29 09:46:58 +00:00
vdimir
3292566603
Format bytes in CreatingSetsOnTheFlyTransform logs 2022-08-29 09:46:57 +00:00
vdimir
031aaf3a45
Add Creating/FilterBySetsOnTheFlyStep for full sorting join 2022-08-29 09:46:57 +00:00
vdimir
c5bc7b0a0c
Resize pipeline after full sort join 2022-08-29 09:46:56 +00:00
Azat Khuzhin
f9812d9917 Fix memory leak while pushing to MVs w/o query context (from Kafka/...) 
While pushign to MVs, there is a low-level code that create
ThreadGroupStatus/ThreadStatus, it is required to gather some metrics
for system.query_views_log.

But, one should not use ThreadGroupStatus of the MainThreadStatus, since
this structure can hold some state, that may not be cleaned, plus this
may be racy, instead it is better to create new ThreadGroupStatus and
attach it instead.

Also this place misses detachQuery(), and because of this it leaks
ThreadGroupStatus::finished_threads_counters_memory. But it is only the
problem pushing to MVs is done w/o query context (i.e. from Kafka/...),
since when it has query context detachQuery() will be called eventually.

Before this patch series, when I've tried the reproducer with
500 MVs attached to Kafka engine (that @den-crane suggested), jemalloc
report looks like this:

    $ ../jeprof --text ~/ch/tmp/upstream/clickhouse-binary --base jeprof.44384.0.i0.heap jeprof.44384.167.i167.heap
    Using local file /home/azat/ch/tmp/upstream/clickhouse-binary.
    Using local file jeprof.44384.167.i167.heap.
    Total: 915.6 MB
       910.7  99.5%  99.5%    910.7  99.5% Snapshot (inline)
         9.5   1.0% 100.5%      9.5   1.0% std::__1::__libcpp_operator_new (inline)
         0.5   0.1% 100.6%      0.5   0.1% DB::TasksStatsCounters::create

And with focus to this place:

    $ ../jeprof --focus Snapshot --text ~/ch/tmp/upstream/clickhouse-binary --base jeprof.44384.0.i0.heap jeprof.44384.167.i167.heap
    Using local file /home/azat/ch/tmp/upstream/clickhouse-binary.
    Using local file jeprof.44384.167.i167.heap.
    Total: 915.6 MB
       910.7 100.0% 100.0%    910.7 100.0% Snapshot (inline)
         0.0   0.0% 100.0%    910.7 100.0% DB::QueryPipeline::reset
         0.0   0.0% 100.0%    910.7 100.0% DB::StorageKafka::streamToViews
         0.0   0.0% 100.0%    910.7 100.0% DB::StorageKafka::threadFunc
         0.0   0.0% 100.0%    910.7 100.0% ProfileEvents::Counters::getPartiallyAtomicSnapshot
         0.0   0.0% 100.0%    910.7 100.0% ~ThreadStatus
         0.0   0.0% 100.0%    910.7 100.0% ~ViewRuntimeData
         0.0   0.0% 100.0%    910.7 100.0% ~ViewRuntimeStats (inline)

Actually this report does not looks great (you understand it because I
stripped it), because --text does not that smart, but if you will use
--pdf for the report you will see the stacktrace (will attach pdf to the
pull request).

But after this patch series the process RSS does not goes beyond
~700MiB.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-08-29 11:36:33 +02:00
Amos Bird
15a69bce84
Use index when row_policy_filter is always false 2022-08-29 16:44:32 +08:00
Alexey Milovidov
365a600fdb Merge branch 'force-documentation-3' of github.com:ClickHouse/ClickHouse into force-documentation-3 2022-08-27 22:28:54 +02:00
Alexey Milovidov
6b2e227c8b Fix integration test 2022-08-27 22:28:38 +02:00
Vladimir C
e067629e0d
Merge pull request #40239 from vdimir/vdimir/tmp-file-metrics 
More metrics for on-disk temporary data
 2022-08-26 11:28:01 +02:00
Alexander Gololobov
6a69e08799
Merge pull request #40559 from ClickHouse/lwd_vertical_merge_fix 
Fix vertical merge of parts with lightweight deleted rows
 2022-08-25 20:47:44 +02:00
Frank Chen
bb00dcc19b Remove using namespace from header 
Signed-off-by: Frank Chen <frank.chen021@outlook.com>
 2022-08-25 20:20:13 +08:00
Frank Chen
99c37ce6c6
Merge branch 'master' into tracing_context_propagation 2022-08-25 10:07:16 +08:00
Nikita Taranov
ac34a17551
Merge branch 'master' into control_block_size_in_aggregator 2022-08-24 20:25:28 +02:00
vdimir
91788f29e8
Upd TemporaryFileOnDisk 2022-08-24 16:15:54 +00:00
vdimir
7194df1184
Move back TemporaryFile -> TemporaryFileOnDisk 2022-08-24 16:14:11 +00:00
vdimir
0349c85017
Use getCompressedBytes in BufferingToFileTransform and TemporaryFileStream 2022-08-24 16:14:10 +00:00
vdimir
51c44424cc
More metrics for temp files 2022-08-24 16:14:09 +00:00
vdimir
1321ac87b5
Minor fixes 2022-08-24 16:14:07 +00:00
vdimir
7e0c9062c7
Add ProfileEvents::ExternalSort(Un)CompressedBytes 2022-08-24 16:14:07 +00:00
Kruglov Pavel
e6e7f5db93
Merge pull request #40491 from mini4/fix-settings-input_format_tsv_skip_first_lines 
Fix bug in settings input_format_tsv_skip_first_lines of format TSV
 2022-08-24 15:57:45 +02:00
Alexander Gololobov
1c2dd50ca5 Fix vertical merge of parts with lightweight deleted rows 2022-08-24 15:18:33 +02:00
Kruglov Pavel
0781e8b4f7
Merge pull request #40534 from Avogar/nested-in-avro 
Support reading Array(Record) into flatten nested table in Avro
 2022-08-24 13:33:12 +02:00
Frank Chen
cd19366b44 Move classes into DB::OpenTelemetry namespace 2022-08-24 16:41:40 +08:00
kgurjev
f62c2c3221 Fix bug in settings input_format_tsv_skip_first_lines of format TSV 2022-08-24 10:02:57 +03:00
Kruglov Pavel
72f02bd6eb
Merge pull request #40414 from Avogar/improve-schema-inference-cache 
Improve schema inference cache, respect format settings that can change the schema
 2022-08-23 17:04:58 +02:00
avogar
29a887578b Fix 2022-08-23 11:42:57 +00:00
avogar
581e569d04 Support reading Array(Record) into flatten nested table in Avro 2022-08-23 11:05:02 +00:00
Arthur Passos
f8e2ab0a20 Use FileReader::GetRecordBatchReader instead of FileReader::ReadRowGroup to parse Parquet 2022-08-22 08:21:32 -03:00
Alexey Milovidov
ab91c99495
Merge branch 'master' into control_block_size_in_aggregator 2022-08-20 21:28:27 +03:00
Alexey Milovidov
74e1f4dc61 Fix clang-tidy 2022-08-20 17:09:20 +02:00
avogar
612ffaffde Make schema inference cache better, respect format settings that can change the schema 2022-08-19 16:39:13 +00:00
Nikita Taranov
1b6e7b9ca2
Merge branch 'master' into sort_mode_rename 2022-08-19 12:31:59 +02:00
Kruglov Pavel
b67cb9e378
Merge pull request #40173 from Avogar/arrow-dict 
Improve and fix dictionaries in Arrow format
 2022-08-18 20:54:55 +02:00
Kruglov Pavel
09a2ff8843
Merge pull request #40293 from joshuataylor/feature/arrow-large-binary-string 
Add support for LARGE_BINARY/LARGE_STRING with Arrow
 2022-08-18 14:01:58 +02:00
avogar
a6318cecd5 Fix hive test 2022-08-18 11:32:42 +00:00
Nikolai Kochetov
5a85531ef7
Merge pull request #38286 from Avogar/schema-inference-cache 
Add schema inference cache for s3/hdfs/file/url
 2022-08-18 13:07:50 +02:00
Kruglov Pavel
d7056376eb
Merge pull request #40068 from Avogar/schema-inference-hints 
Allow to specify structure hints in schema inference
 2022-08-18 12:19:45 +02:00
Igor Nikonov
6fe8b61345
Merge branch 'master' into sort_mode_rename 2022-08-17 19:19:29 +02:00
Yakov Olkhovskiy
40fd6e189a
call readColumnWithStringData 2022-08-17 09:54:01 -04:00
Kruglov Pavel
19af748737
Fix typo 2022-08-17 14:29:09 +02:00
Kruglov Pavel
00d04456ff
Try reduce code duplication 2022-08-17 14:28:15 +02:00
Vladimir C
b876cc17c9
Merge pull request #39593 from quickhouse/fixexponentialdecaywindowfunctions 
Fixed point of origin for exponential decay window functions to the last value in window
 2022-08-17 14:19:59 +02:00
Igor Nikonov
5ceaeb9e12 Sorting mode renaming 
+ sort mode -> sort scope
+ Stream -> Global
+ Port -> Stream
 2022-08-17 12:19:36 +00:00
avogar
8dd54c043d Merge branch 'master' of github.com:ClickHouse/ClickHouse into schema-inference-cache 2022-08-17 11:47:40 +00:00
Igor Nikonov
46ed4f6cdf
Merge pull request #38719 from ClickHouse/skipping_sorting_step 
SortingStep: deduce way to sort based on input stream sort description
 2022-08-17 12:58:11 +02:00