thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-12 09:22:05 +00:00
Code Issues Packages Projects Releases Wiki Activity
126,736 Commits 9,297 Branches 1,782 Tags 2.1 GiB
c2215cb5f0
Commit Graph

5297 Commits

Author SHA1 Message Date
Robert Schulze
c2215cb5f0
Support string arguments 2023-10-22 09:28:45 +00:00
Michael Kolupaev
ce7eca0615
DWARF input format (#55450) 
* Add ReadBufferFromFileBase::isRegularLocalFile()

* DWARF input format

* Review comments

* Changed things around ENABLE_EMBEDDED_COMPILER build setting

* Added 'ranges' column

* no-msan no-ubsan
 2023-10-16 17:00:07 -07:00
Jiebin Sun
df17cd467b
Release more num_streams if data is small (#53867) 
* Release more num_streams if data is small

Besides the sum_marks and min_marks_for_concurrent_read, we could also involve the
system cores to get the num_streams if the data is small. Increasing the num_streams
and decreasing the min_marks_for_concurrent_read would improve the parallel performance
if the system has plentiful cores.

Test the patch on 2x80 vCPUs system. Q39 of clickbench has got 3.3x performance improvement.
Q36 has got 2.6x performance improvement. The overall geomean has got 9% gain.

Signed-off-by: Jiebin Sun <jiebin.sun@intel.com>

* Release more num_streams if data is small
Change the min marks from 4 to 8 as the profit is small and 8 granules
is the default block size.

Signed-off-by: Jiebin Sun <jiebin.sun@intel.com>

---------

Signed-off-by: Jiebin Sun <jiebin.sun@intel.com>
 2023-10-16 18:41:38 +02:00
alesapin
3b02748cb6 Fix some typos 2023-10-15 15:43:02 +02:00
Alexander Tokmakov
e3e105d154
Merge pull request #55527 from azat/values-eof-check-fix 
Fix checking of non handled data for Values format
 2023-10-13 18:07:02 +02:00
Alexey Milovidov
8a1363bcf1
Merge pull request #49486 from bigo-sg/test_hive_null_as_default 
Set defaults_for_omitted_fields to true for hive text format
 2023-10-13 02:01:09 +02:00
Alexey Milovidov
b53a98217c
Merge pull request #55489 from kitaisreal/external-aggregation-improve-performance 
Improve performance of external aggregation with a lot of temporary files
 2023-10-12 18:00:41 +02:00
Alexey Milovidov
e48fad0e24
Merge pull request #55495 from kitaisreal/aggregating-transform-init-generate-race-condition-fix 
AggregatingTransform initGenerate race condition fix
 2023-10-12 17:54:36 +02:00
Azat Khuzhin
2cbb069b68 Add ability to ignore data after semicolon in Values format 
This is required for client, to handle comments in multiquery mode.

v0: separate context for input format
v2: cannot use separate context since params and stuff are changed in global context
v3: do not sent this setting to the server (breaks queries for readonly profiles)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-10-12 14:55:26 +02:00
Antonio Andelic
22f5d6d708 Reset promise_to_build 2023-10-12 10:57:21 +00:00
Azat Khuzhin
f379d9cac5 Fix checking of non handled data for Values format 
PeekableReadBuffer::hasUnreadData() does not checks the underlying
buffer, and so it simply ignore some issues, like:

    INSERT INTO test_01179_str values ('foo'); ('bar')

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-10-12 12:25:08 +02:00
taiyang-li
7cd24d0af0 solve conflicts 2023-10-12 10:30:05 +08:00
Robert Schulze
7b14392675
Merge pull request #55481 from rschu1ze/better-use-mysql-types-in-show-columns 
Make `use_mysql_types_in_show_columns` affect only `SHOW COLUMNS`
 2023-10-12 00:36:46 +02:00
Maksim Kita
ae94055a29 Updated implementation 2023-10-11 19:01:00 +03:00
Maksim Kita
f9033bdf31 Improve performance of external aggregation with a lot of temporary files 2023-10-11 19:01:00 +03:00
Igor Nikonov
62060a0603 Merge remote-tracking branch 'origin/master' into pr-coordinator-usage-cleanup 2023-10-11 15:11:15 +00:00
Robert Schulze
97d8e16e8d
Fix MySQL packet row data type 
Cf. https://github.com/ClickHouse/ClickHouse/pull/55479#discussion_r1355175581
 2023-10-11 15:09:50 +00:00
Igor Nikonov
9d95f4e1b6 Cleanup: parallel replica coordinator usage 2023-10-11 15:04:59 +00:00
robot-ch-test-poll1
80aa9cad51
Merge pull request #55479 from slvrtrn/fix-mysql-datetime-incompatibilities 
Fix MySQL text protocol DateTime formatting and LowCardinality(Nullable(T)) types reporting
 2023-10-11 16:56:26 +02:00
Maksim Kita
0859fc8de9 AggregatingTransform initGenerate race condition fix 2023-10-11 14:28:22 +03:00
slvrtrn
e06d3ca1a5 Fix MySQL text protocol DateTime 
Introduce `removeLowCardinalityAndNullable` function
Fix incorrect removeLowCar/removeNullable usages
Add more MySQL text protocol tests
Deprecate old Java client tests
Use JDK 17 for test MySQL Java container
 2023-10-10 19:51:09 +02:00
Nikita Mikhaylov
4456fe40f9
Remove the old code for projection analysis (#55112) 2023-10-10 17:13:32 +02:00
Alexey Milovidov
e204b1d616
Merge pull request #55419 from evillique/mongo-fix 
Fix MongoDB connection issues
 2023-10-10 01:30:04 +02:00
Nikolay Degterinsky
153b542b29 Small fix 2023-10-09 15:13:45 +00:00
Nikolay Degterinsky
a1e56a3df1 Fix MongoDB connection issues 2023-10-09 14:46:48 +00:00
Alexey Milovidov
e0258582ae
Merge pull request #55338 from azat/fix-CreatingSetsTransform-data-race 
Fix data-race in CreatingSetsTransform (on errors) due to throwing shared exception
 2023-10-09 13:27:10 +02:00
Alexey Milovidov
fac83efdd2
Merge pull request #55335 from azat/fix-sort-sigsegv 
Fix SIGSEGV in MergeSortingPartialResultTransform (due to zero chunks after remerge())
 2023-10-09 02:48:12 +02:00
Azat Khuzhin
56dbba1f95 Fix SIGSEGV in MergeSortingPartialResultTransform (due to zero chunks after remerge()) 
It is possible to have non zero input chunks before remerge() and zero
after:

    2023.10.08 10:21:20.944928 [ 4321 ] {427df456-1400-4fbe-8bd7-c4de139f00ca} <Debug> MergeSortingTransform: Re-merging intermediate ORDER BY data (1 blocks with 0 rows) to save memory consumption
    2023.10.08 10:36:16.447001 [ 14466 ] {} <Fatal> BaseDaemon: (version 23.10.1.1, build id: 3E53B6B3A53FD562F44C88703BD88EB713881A44, git hash: 5ddfb170a1096cf88664b1a4b9b7bd2e7ef36c29) (from thread 4321) (query_id: 427df456-1400-4fbe-8bd7-c4de139f00ca) (query: SELECT CounterID FROM test.visits ARRAY JOIN Goals.ID WHERE CounterID = 942285 ORDER BY CounterID

CI: https://s3.amazonaws.com/clickhouse-test-reports/55276/099665478df3e77d1df0332dd705ca7209e903af/stress_test__debug_.html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-10-08 22:21:01 +02:00
Alexey Milovidov
fa54b21424
Merge pull request #55103 from ClickHouse/fix_shell_command_abort 
Proper cleanup in case of exception in ctor of ShellCommandSource
 2023-10-08 21:36:28 +02:00
Azat Khuzhin
cd3c140dd6 Fix data-race in CreatingSetsTransform (on errors) due to throwing shared exception 
WARNING: ThreadSanitizer: data race (pid=3436)
  Read of size 1 at 0x7b50001b07a7 by thread T187:
    0 std::__1::basic_string<>::__is_long[abi:v15000] const build_docker/./contrib/llvm-project/libcxx/include/string:1499:33 (clickhouse-tsan+0x1faa29ff) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    1 std::__1::basic_string<>::size[abi:v15000]() const build_docker/./contrib/llvm-project/libcxx/include/string:968:17 (clickhouse-tsan+0x1faa29ff)
    2 std::__1::basic_string<>::empty[abi:v15000]() const build_docker/./contrib/llvm-project/libcxx/include/string:996:42 (clickhouse-tsan+0x1faa29ff)
    3 Poco::Exception::extendedMessage() build_docker/./base/poco/Foundation/src/Exception.cpp:114:13 (clickhouse-tsan+0x1faa29ff)
    4 DB::Exception::addMessage() <null> (clickhouse-tsan+0x7e0d5f1) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    5 DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:70:23 (clickhouse-tsan+0x1b3f2fed) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    6 DB::ExecutionThreadContext::executeTask() build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:95:9 (clickhouse-tsan+0x1b3f2fed)
    7 DB::PipelineExecutor::executeStepImpl(unsigned long, std::__1::atomic<bool>*) build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273:26 (clickhouse-tsan+0x1b3e5a10) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    8 DB::PipelineExecutor::executeSingleThread(unsigned long) build_docker/./src/Processors/Executors/PipelineExecutor.cpp:239:5 (clickhouse-tsan+0x1b3e4bb8) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    9 DB::PipelineExecutor::executeImpl(unsigned long, bool) build_docker/./src/Processors/Executors/PipelineExecutor.cpp:399:9 (clickhouse-tsan+0x1b3e4bb8)
    10 DB::PipelineExecutor::execute(unsigned long, bool) build_docker/./src/Processors/Executors/PipelineExecutor.cpp:111:9 (clickhouse-tsan+0x1b3e48be) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    11 DB::CompletedPipelineExecutor::execute() build_docker/./src/Processors/Executors/CompletedPipelineExecutor.cpp:110:18 (clickhouse-tsan+0x1b3e31c8) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    12 DB::FutureSetFromSubquery::buildOrderedSetInplace(std::__1::shared_ptr<DB::Context const> const&) build_docker/./src/Interpreters/PreparedSets.cpp:202:14 (clickhouse-tsan+0x19a58396) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)

  Previous write of size 8 at 0x7b50001b07a0 by thread T192:
    0 std::__1::basic_string<>::__set_long_cap[abi:v15000](unsigned long) build_docker/./contrib/llvm-project/libcxx/include/string:1592:37 (clickhouse-tsan+0x238809e2) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    1 std::__1::basic_string<>::__grow_by_and_replace(unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, char const*) build_docker/./contrib/llvm-project/libcxx/include/string:2367:5 (clickhouse-tsan+0x238809e2)
    2 std::__1::basic_string<>::append(char const*, unsigned long) build_docker/./contrib/llvm-project/libcxx/include/string:2692:9 (clickhouse-tsan+0x238809e2)
    3 std::__1::basic_string<>::append(char const*) build_docker/./contrib/llvm-project/libcxx/include/string:2849:12 (clickhouse-tsan+0x1faa2a58) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    4 Poco::Exception::extendedMessage() build_docker/./base/poco/Foundation/src/Exception.cpp:114:27 (clickhouse-tsan+0x1faa2a58)
    5 DB::Exception::addMessage() <null> (clickhouse-tsan+0x7e0d5f1) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    6 DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:70:23 (clickhouse-tsan+0x1b3f2fed) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    7 DB::ExecutionThreadContext::executeTask() build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:95:9 (clickhouse-tsan+0x1b3f2fed)
    8 DB::PipelineExecutor::executeStepImpl(unsigned long, std::__1::atomic<bool>*) build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273:26 (clickhouse-tsan+0x1b3e5a10) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    9 DB::PipelineExecutor::executeSingleThread(unsigned long) build_docker/./src/Processors/Executors/PipelineExecutor.cpp:239:5 (clickhouse-tsan+0x1b3e4bb8) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    10 DB::PipelineExecutor::executeImpl(unsigned long, bool) build_docker/./src/Processors/Executors/PipelineExecutor.cpp:399:9 (clickhouse-tsan+0x1b3e4bb8)
    11 DB::PipelineExecutor::execute(unsigned long, bool) build_docker/./src/Processors/Executors/PipelineExecutor.cpp:111:9 (clickhouse-tsan+0x1b3e48be) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    12 DB::CompletedPipelineExecutor::execute() build_docker/./src/Processors/Executors/CompletedPipelineExecutor.cpp:110:18 (clickhouse-tsan+0x1b3e31c8) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    13 DB::FutureSetFromSubquery::buildOrderedSetInplace(std::__1::shared_ptr<DB::Context const> const&) build_docker/./src/Interpreters/PreparedSets.cpp:202:14 (clickhouse-tsan+0x19a58396) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)

  Location is heap block of size 480 at 0x7b50001b0600 allocated by thread T193:
    0 posix_memalign <null> (clickhouse-tsan+0x7d6ac85) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    1 std::__1::__libcpp_aligned_alloc[abi:v15000](unsigned long, unsigned long) build_docker/./contrib/llvm-project/libcxxabi/../libcxx/include/new:328:9 (clickhouse-tsan+0x238b5a05) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    2 __cxxabiv1::__aligned_malloc_with_fallback(unsigned long) build_docker/./contrib/llvm-project/libcxxabi/src/fallback_malloc.cpp:215:20 (clickhouse-tsan+0x238b5a05)
    3 __cxa_allocate_exception build_docker/./contrib/llvm-project/libcxxabi/src/cxa_exception.cpp:190:17 (clickhouse-tsan+0x238b31e5) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    4 std::exception_ptr std::make_exception_ptr[abi:v15000]<DB::Exception>(DB::Exception) build_docker/./contrib/llvm-project/libcxx/include/exception:206:9 (clickhouse-tsan+0x144bfa8a) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    5 DB::CreatingSetsTransform::~CreatingSetsTransform() build_docker/./src/Processors/Transforms/CreatingSetsTransform.cpp:30:45 (clickhouse-tsan+0x1b7603c3) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    ...
    35 DB::FutureSetFromSubquery::buildOrderedSetInplace(std::__1::shared_ptr<DB::Context const> const&) build_docker/./src/Interpreters/PreparedSets.cpp:211:1 (clickhouse-tsan+0x19a58608) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
 2023-10-08 17:16:39 +02:00
Azat Khuzhin
099665478d Fix incorrect merging of Nested for SELECT FINAL FROM SummingMergeTree 
The problem was the order of the columns, in case of SELECT FINAL it got
"counters_Map.count", "counters_Map.id"

But in case of OPTIMIZE FINAL it got "counters_Map.id",
"counters_Map.count" correctly.

Note, that this bugs exists there from the very recent versions, I've
checked 19.x and it was there.

P.S. there is a workaround for this problem, if you will use one of the
following patterns for key columns:
- *ID
- *Key
- *Type

That way it will be explicitly matched as key and everything will work.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-10-08 07:32:47 +02:00
vdimir
3a9abde35d
Merge pull request #54514 from ClickHouse/vdimir/allow_experimental_partial_result 
Add setting allow_experimental_partial_result
 2023-09-29 10:32:57 +02:00
Alexander Gololobov
d81d5dd75c Cleanup in case of exception in ctor 2023-09-28 16:28:35 +02:00
Nikita Taranov
0e506b618e
impl (#54934) 2023-09-28 14:12:19 +02:00
vdimir
a134569bc3
fix MergeSortingPartialResultTransform::getRealProcessorSnapshot 2023-09-28 09:41:01 +00:00
vdimir
4e86eec7d1
fix Pipe::unitePipes with partial results 2023-09-28 09:41:00 +00:00
vdimir
3f3feea0b7
Add setting allow_experimental_partial_result 2023-09-28 09:40:56 +00:00
robot-clickhouse-ci-1
7a825c1417
Merge pull request #54427 from Avogar/json-object-as-tuple-inference 
Add new features to schema inference for JSON formats
 2023-09-27 20:12:45 +02:00
Robert Schulze
2d750e9e2f
Merge remote-tracking branch 'rschu1ze/master' into clang-tidy-reenable-checks 2023-09-27 12:15:29 +00:00
Antonio Andelic
580aa0031c
Merge pull request #55013 from ClickHouse/another-promise-fix 
Set exception for promise in `CreatingSetsTransform` in more cases
 2023-09-27 09:01:42 +02:00
Michael Kolupaev
23bc286280
Print more information about one logical error in MergeTreeDataWriter (#55001) 2023-09-26 13:23:26 -07:00
Robert Schulze
cde10fe7b5
Merge remote-tracking branch 'rschu1ze/master' into clang-tidy-reenable-checks 2023-09-26 18:59:41 +00:00
Kruglov Pavel
bea80ab5b7
Merge branch 'master' into json-object-as-tuple-inference 2023-09-26 15:23:08 +02:00
Kruglov Pavel
69a17bbef6
Merge pull request #52853 from Avogar/http-valid-json-on-exception 
Output valid JSON/XML on excetpion during HTTP query execution
 2023-09-26 14:25:55 +02:00
Robert Schulze
9fff447716
Re-enable clang-tidy checks 2023-09-26 09:34:12 +00:00
Antonio Andelic
098efb261e Throw proper exception if promise was not set 2023-09-26 09:03:38 +00:00
avogar
9e75825515 Merge branch 'master' of github.com:ClickHouse/ClickHouse into json-object-as-tuple-inference 2023-09-25 17:24:36 +00:00
avogar
42ca897f2d Better schema inference for JSON formats 2023-09-25 15:42:59 +00:00
robot-ch-test-poll4
9d61ed4665
Merge pull request #54664 from bigo-sg/arrow_join_max_block_size 
Respect max_block_size for array join to avoid possible OOM
 2023-09-23 21:30:51 +02:00
robot-ch-test-poll4
ba6f0431a5
Merge pull request #54933 from ClibMouse/feature/big-endian-bson-each-row 
Provide support for BSON on BE
 2023-09-23 03:00:27 +02:00