thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-11 08:52:06 +00:00
Code Issues Packages Projects Releases Wiki Activity
131,051 Commits 9,280 Branches 1,782 Tags 2.1 GiB
e09c68db32
Commit Graph

5564 Commits

Author SHA1 Message Date
Nikolai Kochetov
4deaf7cefb
Merge pull request #56134 from yariks5s/force_optimize_projection_name 
Implementing force_optimize_projection_name
 2023-11-01 13:12:10 +01:00
taiyang-li
24c45a4ee0 fix failed uts 2023-11-01 18:47:11 +08:00
taiyang-li
001cbe7912 fix typos 2023-11-01 16:58:25 +08:00
taiyang-li
b276587422 fix failed uts 2023-11-01 15:43:20 +08:00
lgbo-ustc
8334585eaf improve parquet struct field reading 2023-11-01 15:18:39 +08:00
Alex Cheng
9e31858100
Merge branch 'ClickHouse:master' into refactor-MergeJoinTransform 2023-11-01 09:32:28 +08:00
yariks5s
6c4bf59021 fix suggestions and enhance tests 2023-10-31 18:10:55 +00:00
avogar
518e52473d Fix schema cache for fallback JSON->JSONEachRow with changed settings 2023-10-31 14:12:38 +00:00
Kruglov Pavel
4faa3d0294
Revert "Revert "Fix output/input of Arrow dictionary column"" 2023-10-31 12:30:45 +01:00
Nikolai Kochetov
f748f12426
Merge pull request #51746 from ClickHouse/fix-read-in-order-with-array-join 
Fix 'Cannot find column' in read-in-order optimization with ARRAY JOIN
 2023-10-31 11:51:01 +01:00
Nikolai Kochetov
0523e6cbd0
Merge pull request #53282 from ClickHouse/planner-prepare-filters-for-analysis-2 
Support skip_unused_shards in Analyzer
 2023-10-31 11:33:59 +01:00
Alexey Milovidov
467b4d85e2
Revert "Fix output/input of Arrow dictionary column" 2023-10-31 09:28:09 +03:00
taiyang-li
c97b2c5be7 fix code style 2023-10-31 12:00:45 +08:00
taiyang-li
e5db57204d fix bugs 2023-10-31 11:57:47 +08:00
taiyang-li
b72341e1a8 Merge branch 'master' into orc_tuple_field_prune 2023-10-31 10:07:43 +08:00
Han Fei
1237cef26d address comments 2023-10-31 00:53:46 +01:00
Kruglov Pavel
4c2a132d96
Merge pull request #55989 from Avogar/lc-as-arrow-dict-fix 
Fix output/input of Arrow dictionary column
 2023-10-30 20:47:49 +01:00
Kruglov Pavel
4effc676f9
Merge pull request #56046 from Avogar/cr-in-unquoted-csv-string 
Allow unquoted strings with CR in CSV format
 2023-10-30 20:46:20 +01:00
Nikolai Kochetov
84f6a243b7 Merge branch 'master' into fix-read-in-order-with-array-join 2023-10-30 16:35:31 +00:00
yariks5s
03236c48ed init 2023-10-30 16:21:50 +00:00
yariks5s
9a2d89e3e4 removed getSize() and enhanced docs 2023-10-30 12:42:19 +00:00
taiyang-li
ad67b6c2ea allow tuple field pruning 2023-10-30 19:33:06 +08:00
Nikolai Kochetov
554ceb4e1d Merge branch 'master' into planner-prepare-filters-for-analysis-2 2023-10-30 11:56:30 +01:00
Maksim Kita
f8e209ebd2 WindowTransform decrease amount of virtual function calls 2023-10-30 13:50:25 +03:00
Kruglov Pavel
c10a3b3838
Merge branch 'master' into lc-as-arrow-dict-fix 2023-10-30 11:20:57 +01:00
avogar
57bc4854c2 Fix 2023-10-30 10:17:49 +00:00
avogar
d1fcbc6e47 Fix fetching schema from schema registry in AvroConfluent 2023-10-30 10:17:48 +00:00
Han Fei
19cdc6dfe2 Merge branch 'master' into hanfei/statistic 2023-10-30 04:53:18 +01:00
yariks5s
e14a7f066a fix typos 2023-10-28 01:46:59 +00:00
yariks5s
894724bfb3 suggested changes 2023-10-28 01:17:25 +00:00
yariks5s
23635352f1 fixed due to review 2023-10-27 15:43:03 +00:00
Kruglov Pavel
bb4b95e891
Merge branch 'master' into schema-inference-union 2023-10-27 14:53:58 +02:00
frinkr
18c50c11b3
Multithreading after window functions (#50771) 
* feat: Preserve number of streams after evaluation the window functions to allow parallel stream processing

* fix style

* fix style

* fix style

* setting query_plan_preserve_num_streams_after_window_functions default true

* fix tests by SETTINGS query_plan_preserve_num_streams_after_window_functions=0

* fix test references

* Resize the streams after the last window function, to keep the order between WindowTransforms (and WindowTransform works on single stream anyway).

* feat: Preserve number of streams after evaluation the window functions to allow parallel stream processing

* fix style

* fix style

* fix style

* setting query_plan_preserve_num_streams_after_window_functions default true

* fix tests by SETTINGS query_plan_preserve_num_streams_after_window_functions=0

* fix test references

* Resize the streams after the last window function, to keep the order between WindowTransforms (and WindowTransform works on single stream anyway).

* add perf test

* perf: change the dataset from 50M to 5M

* rename query_plan_preserve_num_streams_after_window_functions -> query_plan_enable_multithreading_after_window_functions

* update test reference

* fix clang-tidy

---------

Co-authored-by: Nikita Taranov <nikita.taranov@clickhouse.com>
 2023-10-27 12:36:28 +02:00
Kruglov Pavel
570b66f027
Merge branch 'master' into schema-inference-union 2023-10-26 19:26:00 +02:00
avogar
9d207bf027 Allow unquoted strings with CR in CSV format 2023-10-26 13:50:54 +00:00
zvonand
0766c73aab Rename date_time_overflow_mode -> date_time_overflow_behavior, moved it to format settings 2023-10-25 23:11:13 +02:00
zvonand
5b86e8c714 updated after review 2023-10-25 23:10:58 +02:00
zvonand
2f3695add8 Introduce setting for dt overflow exception 
Added tests and docs
 2023-10-25 23:10:24 +02:00
Alexey Milovidov
bb5a60dc19
Merge pull request #55893 from ClickHouse/revert-partial-result-2 
Revert "Revert "Revert "Add settings for real-time updates during query execution"""
 2023-10-25 22:20:28 +02:00
Igor Nikonov
1a5d485c6c
Merge branch 'master' into pr-progress-bar 2023-10-25 19:53:20 +02:00
yariks5s
2ab1ae42c1 added docs and tests, style check 2023-10-25 10:37:05 +00:00
Igor Nikonov
fafe452f1e
Merge branch 'master' into pr-cleanup-remote-source 2023-10-25 11:48:23 +02:00
robot-ch-test-poll1
ef78889aa2
Merge pull request #55952 from ClickHouse/disable_apply_deleted_mask 
Added a setting to allow reading rows marked as deleted
 2023-10-25 01:14:07 +02:00
Igor Nikonov
7b7f100250 Merge remote-tracking branch 'origin/master' into pr-progress-bar 2023-10-24 21:02:29 +00:00
Igor Nikonov
c3bd80f1c8 Set progress callback in constructor 2023-10-24 21:02:20 +00:00
avogar
c080ba9d7e Fix output LowCardinality as Arrow dictionary 2023-10-24 19:49:04 +00:00
李扬
465962df7f
Support orc filter push down (file + stripe + rowgroup level) (#55330) 
* support orc filter push down

* update orc lib version

* replace setqueryinfo with setkeycondition

* fix issue https://github.com/ClickHouse/ClickHouse/issues/53536

* refactor source with key condition

* fix building error

* remove std::cout

* update orc

* update orc version

* fix bugs

* improve code

* upgrade orc lib

* fix code style

* change as requested

* add performance tests for orc filter push down

* add performance tests for orc filter push down

* fix all bugs

* fix default as null issue

* add uts for null as default issues

* upgrade orc lib

* fix failed orc lib uts and fix typo

* fix failed uts

* fix failed uts

* fix ast fuzzer tests

* fix bug of uint64 overflow in https://s3.amazonaws.com/clickhouse-test-reports/55330/de22fdcaea2e12c96f300e95f59beba84401712d/fuzzer_astfuzzerubsan/report.html

* fix asan fatal caused by reused column vector batch in native orc input format. refer to https://s3.amazonaws.com/clickhouse-test-reports/55330/be39d23af2d7e27f5ec7f168947cf75aeaabf674/stateless_tests__asan__[4_4].htm

* fix wrong performance tests

* disable 02892_orc_filter_pushdown on aarch64. https://s3.amazonaws.com/clickhouse-test-reports/55330/be39d23af2d7e27f5ec7f168947cf75aeaabf674/stateless_tests__aarch64_.html

* add some comments

* add some comments

* inline range::equals and range::less

* fix data race of key condition

* trigger ci
 2023-10-24 12:08:17 -07:00
yariks5s
4e09fb3e27 made_logic 2023-10-24 14:55:52 +00:00
Igor Nikonov
2c055480d6 Remove unnecessary flag 2023-10-24 14:52:47 +00:00
Kruglov Pavel
9a56ec4d63
Merge pull request #55891 from Avogar/try-fix-orc 
Try to fix possible segfault in Native ORC input format
 2023-10-24 13:57:43 +02:00
Alexander Gololobov
959b8b64bd Added a setting to allow reading rows marked as deleted 2023-10-23 19:59:17 +02:00
Robert Schulze
bef5cea5a0
Merge remote-tracking branch 'rschu1ze/master' into string-addinterval 2023-10-23 14:08:18 +00:00
Igor Nikonov
0bbf7ba921
Merge branch 'master' into pr-progress-bar 2023-10-23 14:56:59 +02:00
Kruglov Pavel
32da07b4ee
Merge pull request #55695 from Avogar/restrict-iceberg 
Fixes and improvements for Iceberg storage
 2023-10-23 13:58:23 +02:00
Alex Cheng
49b3136491
Merge branch 'ClickHouse:master' into refactor-MergeJoinTransform 2023-10-23 09:56:22 +08:00
Robert Schulze
c2215cb5f0
Support string arguments 2023-10-22 09:28:45 +00:00
Alexey Milovidov
5217d64551 Remove garbage 2023-10-22 01:53:50 +02:00
Alexey Milovidov
1be4ff229b
Merge pull request #55895 from ClickHouse/fix-window-functions-with-sparse-columns 
Fix window functions in case of sparse columns.
 2023-10-22 00:02:34 +02:00
János Benjamin Antal
1b0cadebd0 Convert sparse columns to full for window functions 2023-10-21 11:46:25 +00:00
Igor Nikonov
e378d920c4
Merge branch 'master' into pr-progress-bar 2023-10-21 13:16:12 +02:00
Alexey Milovidov
7ec4b99e94 Revert partial result 2023-10-21 03:14:22 +02:00
Kruglov Pavel
6f61ccfe28
Merge branch 'master' into schema-inference-union 2023-10-20 22:54:11 +02:00
avogar
6934e27e8b Add union mode for schema inference to infer union schema of files with different schemas 2023-10-20 20:46:41 +00:00
avogar
8cc0dc17eb Try to fix possible segfault in Native ORC input format 2023-10-20 18:50:48 +00:00
yariks5s
87f26f5132 dealt with 2dim arrays 2023-10-20 17:05:05 +00:00
Anton Popov
5819bcd07a
Support asynchronous inserts for native protocol (#54730) 
* support async insert for native protocol

* use separate queue for async inserts via native protocol

* fix test

* better logging for async inserts and more tests

* disable mixed internal and external data in async inserts

* fix tests

* fix quota in async inserts

* disable async insert for secondary query of distributed
 2023-10-20 18:39:48 +02:00
Raúl Marín
d08ce13107
Merge pull request #51692 from Algunenano/parallel_replicas_row_estimation 
Automatic decision of nº of parallel replicas
 2023-10-20 15:50:43 +02:00
Azat Khuzhin
d5c960ceb6 Fix data race in CreatingSetsTransform 
The problem is that findOrPromiseToBuild() can also call
shared_future::get():

    (lldb) bt 10
    * thread 194, name = 'MergeMutate', stop reason = step over
      * frame 0: 0x0000564126046bd2 clickhouse`std::rethrow_exception(p=(__ptr_ = 0x00007b5000130c80)) at exception_pointer_cxxabi.ipp:68:39
        frame 1: 0x000056411c4c5ddd clickhouse`std::__1::__assoc_state<std::__1::shared_ptr<DB::Set>>::copy(this=<unavailable>) at future:701:9
        frame 2: 0x000056411c4c0e3f clickhouse`DB::PreparedSetsCache::findOrPromiseToBuild(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) [inlined] std::__1::shared_future<std::__1::shared_ptr<DB::Set>>::get[abi:v15000](this=0x00007b0c00154eb8) const at future:2243:46
        frame 3: 0x000056411c4c0e2e clickhouse`DB::PreparedSetsCache::findOrPromiseToBuild(this=0x00007b1c00077018, key="__set_10438532338982543705_15680061365857338647") at PreparedSets.cpp:349:116

Note, that my test did not find the problem because of #55261 had not
been merged yet.

Follow-up for: #55338
Fixes: #55279
v2: s/createDeprecated/createRuntime/

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-10-19 12:26:02 +02:00
yariks5s
6dc88a4ca4 new changes 2023-10-18 18:02:05 +00:00
Dmitry Novik
5ca43e51c1 Remove unused code 2023-10-17 20:45:00 +00:00
avogar
323486f9e8 Add tests 2023-10-17 18:10:47 +00:00
Han Fei
4b23142775
Merge branch 'master' into hanfei/statistic 2023-10-17 16:40:31 +02:00
Raúl Marín
3eaf752284 Merge remote-tracking branch 'blessed/master' into parallel_replicas_row_estimation 2023-10-17 11:36:39 +02:00
Alex Cheng
ce493b57f0
Merge branch 'ClickHouse:master' into refactor-MergeJoinTransform 2023-10-17 16:42:53 +08:00
Igor Nikonov
4b36761f89
Merge branch 'master' into pr-progress-bar 2023-10-17 10:36:32 +02:00
Michael Kolupaev
ce7eca0615
DWARF input format (#55450) 
* Add ReadBufferFromFileBase::isRegularLocalFile()

* DWARF input format

* Review comments

* Changed things around ENABLE_EMBEDDED_COMPILER build setting

* Added 'ranges' column

* no-msan no-ubsan
 2023-10-16 17:00:07 -07:00
Jiebin Sun
df17cd467b
Release more num_streams if data is small (#53867) 
* Release more num_streams if data is small

Besides the sum_marks and min_marks_for_concurrent_read, we could also involve the
system cores to get the num_streams if the data is small. Increasing the num_streams
and decreasing the min_marks_for_concurrent_read would improve the parallel performance
if the system has plentiful cores.

Test the patch on 2x80 vCPUs system. Q39 of clickbench has got 3.3x performance improvement.
Q36 has got 2.6x performance improvement. The overall geomean has got 9% gain.

Signed-off-by: Jiebin Sun <jiebin.sun@intel.com>

* Release more num_streams if data is small
Change the min marks from 4 to 8 as the profit is small and 8 granules
is the default block size.

Signed-off-by: Jiebin Sun <jiebin.sun@intel.com>

---------

Signed-off-by: Jiebin Sun <jiebin.sun@intel.com>
 2023-10-16 18:41:38 +02:00
Raúl Marín
f799f5d7a1 Merge remote-tracking branch 'blessed/master' into parallel_replicas_row_estimation 2023-10-16 15:07:11 +02:00
Igor Nikonov
fdd7bfbb83 Merge remote-tracking branch 'origin/master' into pr-progress-bar 2023-10-16 12:59:00 +00:00
alesapin
3b02748cb6 Fix some typos 2023-10-15 15:43:02 +02:00
Alexander Tokmakov
e3e105d154
Merge pull request #55527 from azat/values-eof-check-fix 
Fix checking of non handled data for Values format
 2023-10-13 18:07:02 +02:00
yariks5s
cb08da617f added read and parse impl 2023-10-13 15:16:07 +00:00
Dmitry Novik
d4be706ed4 Merge remote-tracking branch 'origin/master' into parallel-window 2023-10-13 13:20:38 +00:00
Raúl Marín
500dab9569 Merge remote-tracking branch 'blessed/master' into parallel_replicas_row_estimation 2023-10-13 10:12:57 +02:00
Alexey Milovidov
8a1363bcf1
Merge pull request #49486 from bigo-sg/test_hive_null_as_default 
Set defaults_for_omitted_fields to true for hive text format
 2023-10-13 02:01:09 +02:00
Dmitry Novik
cec02c60e8 Merge remote-tracking branch 'origin/master' into parallel-window 2023-10-12 22:45:30 +00:00
Igor Nikonov
b220df93df Remove unnecessary total rows update in parallel replicas 2023-10-12 21:33:25 +00:00
yariks5s
9ae025d7e6 mid commit 2023-10-12 17:37:59 +00:00
Alexey Milovidov
b53a98217c
Merge pull request #55489 from kitaisreal/external-aggregation-improve-performance 
Improve performance of external aggregation with a lot of temporary files
 2023-10-12 18:00:41 +02:00
Alexey Milovidov
e48fad0e24
Merge pull request #55495 from kitaisreal/aggregating-transform-init-generate-race-condition-fix 
AggregatingTransform initGenerate race condition fix
 2023-10-12 17:54:36 +02:00
Dmitry Novik
f31a6f2c0d Support only Full Sort 2023-10-12 14:49:16 +00:00
Han Fei
42dc61abf5 Merge branch 'master' into hanfei/statistic 2023-10-12 16:19:45 +02:00
Azat Khuzhin
2cbb069b68 Add ability to ignore data after semicolon in Values format 
This is required for client, to handle comments in multiquery mode.

v0: separate context for input format
v2: cannot use separate context since params and stuff are changed in global context
v3: do not sent this setting to the server (breaks queries for readonly profiles)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-10-12 14:55:26 +02:00
Antonio Andelic
22f5d6d708 Reset promise_to_build 2023-10-12 10:57:21 +00:00
Azat Khuzhin
f379d9cac5 Fix checking of non handled data for Values format 
PeekableReadBuffer::hasUnreadData() does not checks the underlying
buffer, and so it simply ignore some issues, like:

    INSERT INTO test_01179_str values ('foo'); ('bar')

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-10-12 12:25:08 +02:00
taiyang-li
7cd24d0af0 solve conflicts 2023-10-12 10:30:05 +08:00
Robert Schulze
7b14392675
Merge pull request #55481 from rschu1ze/better-use-mysql-types-in-show-columns 
Make `use_mysql_types_in_show_columns` affect only `SHOW COLUMNS`
 2023-10-12 00:36:46 +02:00
Maksim Kita
ae94055a29 Updated implementation 2023-10-11 19:01:00 +03:00
Maksim Kita
f9033bdf31 Improve performance of external aggregation with a lot of temporary files 2023-10-11 19:01:00 +03:00
Igor Nikonov
62060a0603 Merge remote-tracking branch 'origin/master' into pr-coordinator-usage-cleanup 2023-10-11 15:11:15 +00:00
Robert Schulze
97d8e16e8d
Fix MySQL packet row data type 
Cf. https://github.com/ClickHouse/ClickHouse/pull/55479#discussion_r1355175581
 2023-10-11 15:09:50 +00:00
Igor Nikonov
9d95f4e1b6 Cleanup: parallel replica coordinator usage 2023-10-11 15:04:59 +00:00
robot-ch-test-poll1
80aa9cad51
Merge pull request #55479 from slvrtrn/fix-mysql-datetime-incompatibilities 
Fix MySQL text protocol DateTime formatting and LowCardinality(Nullable(T)) types reporting
 2023-10-11 16:56:26 +02:00
Raúl Marín
0b9bd809e7 Merge remote-tracking branch 'blessed/master' into parallel_replicas_row_estimation 2023-10-11 16:50:22 +02:00
Maksim Kita
0859fc8de9 AggregatingTransform initGenerate race condition fix 2023-10-11 14:28:22 +03:00
slvrtrn
e06d3ca1a5 Fix MySQL text protocol DateTime 
Introduce `removeLowCardinalityAndNullable` function
Fix incorrect removeLowCar/removeNullable usages
Add more MySQL text protocol tests
Deprecate old Java client tests
Use JDK 17 for test MySQL Java container
 2023-10-10 19:51:09 +02:00
Raúl Marín
95d2063e91 Merge remote-tracking branch 'blessed/master' into parallel_replicas_row_estimation 2023-10-10 17:29:45 +02:00
Nikita Mikhaylov
4456fe40f9
Remove the old code for projection analysis (#55112) 2023-10-10 17:13:32 +02:00
Alexey Milovidov
e204b1d616
Merge pull request #55419 from evillique/mongo-fix 
Fix MongoDB connection issues
 2023-10-10 01:30:04 +02:00
Nikolay Degterinsky
153b542b29 Small fix 2023-10-09 15:13:45 +00:00
Nikolay Degterinsky
a1e56a3df1 Fix MongoDB connection issues 2023-10-09 14:46:48 +00:00
Han Fei
fcb19b4f77 Merge branch 'master' into hanfei/statistic 2023-10-09 14:24:58 +02:00
Alexey Milovidov
e0258582ae
Merge pull request #55338 from azat/fix-CreatingSetsTransform-data-race 
Fix data-race in CreatingSetsTransform (on errors) due to throwing shared exception
 2023-10-09 13:27:10 +02:00
Alexey Milovidov
fac83efdd2
Merge pull request #55335 from azat/fix-sort-sigsegv 
Fix SIGSEGV in MergeSortingPartialResultTransform (due to zero chunks after remerge())
 2023-10-09 02:48:12 +02:00
Azat Khuzhin
56dbba1f95 Fix SIGSEGV in MergeSortingPartialResultTransform (due to zero chunks after remerge()) 
It is possible to have non zero input chunks before remerge() and zero
after:

    2023.10.08 10:21:20.944928 [ 4321 ] {427df456-1400-4fbe-8bd7-c4de139f00ca} <Debug> MergeSortingTransform: Re-merging intermediate ORDER BY data (1 blocks with 0 rows) to save memory consumption
    2023.10.08 10:36:16.447001 [ 14466 ] {} <Fatal> BaseDaemon: (version 23.10.1.1, build id: 3E53B6B3A53FD562F44C88703BD88EB713881A44, git hash: 5ddfb170a1096cf88664b1a4b9b7bd2e7ef36c29) (from thread 4321) (query_id: 427df456-1400-4fbe-8bd7-c4de139f00ca) (query: SELECT CounterID FROM test.visits ARRAY JOIN Goals.ID WHERE CounterID = 942285 ORDER BY CounterID

CI: https://s3.amazonaws.com/clickhouse-test-reports/55276/099665478df3e77d1df0332dd705ca7209e903af/stress_test__debug_.html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-10-08 22:21:01 +02:00
Alexey Milovidov
fa54b21424
Merge pull request #55103 from ClickHouse/fix_shell_command_abort 
Proper cleanup in case of exception in ctor of ShellCommandSource
 2023-10-08 21:36:28 +02:00
Azat Khuzhin
cd3c140dd6 Fix data-race in CreatingSetsTransform (on errors) due to throwing shared exception 
WARNING: ThreadSanitizer: data race (pid=3436)
  Read of size 1 at 0x7b50001b07a7 by thread T187:
    0 std::__1::basic_string<>::__is_long[abi:v15000] const build_docker/./contrib/llvm-project/libcxx/include/string:1499:33 (clickhouse-tsan+0x1faa29ff) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    1 std::__1::basic_string<>::size[abi:v15000]() const build_docker/./contrib/llvm-project/libcxx/include/string:968:17 (clickhouse-tsan+0x1faa29ff)
    2 std::__1::basic_string<>::empty[abi:v15000]() const build_docker/./contrib/llvm-project/libcxx/include/string:996:42 (clickhouse-tsan+0x1faa29ff)
    3 Poco::Exception::extendedMessage() build_docker/./base/poco/Foundation/src/Exception.cpp:114:13 (clickhouse-tsan+0x1faa29ff)
    4 DB::Exception::addMessage() <null> (clickhouse-tsan+0x7e0d5f1) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    5 DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:70:23 (clickhouse-tsan+0x1b3f2fed) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    6 DB::ExecutionThreadContext::executeTask() build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:95:9 (clickhouse-tsan+0x1b3f2fed)
    7 DB::PipelineExecutor::executeStepImpl(unsigned long, std::__1::atomic<bool>*) build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273:26 (clickhouse-tsan+0x1b3e5a10) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    8 DB::PipelineExecutor::executeSingleThread(unsigned long) build_docker/./src/Processors/Executors/PipelineExecutor.cpp:239:5 (clickhouse-tsan+0x1b3e4bb8) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    9 DB::PipelineExecutor::executeImpl(unsigned long, bool) build_docker/./src/Processors/Executors/PipelineExecutor.cpp:399:9 (clickhouse-tsan+0x1b3e4bb8)
    10 DB::PipelineExecutor::execute(unsigned long, bool) build_docker/./src/Processors/Executors/PipelineExecutor.cpp:111:9 (clickhouse-tsan+0x1b3e48be) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    11 DB::CompletedPipelineExecutor::execute() build_docker/./src/Processors/Executors/CompletedPipelineExecutor.cpp:110:18 (clickhouse-tsan+0x1b3e31c8) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    12 DB::FutureSetFromSubquery::buildOrderedSetInplace(std::__1::shared_ptr<DB::Context const> const&) build_docker/./src/Interpreters/PreparedSets.cpp:202:14 (clickhouse-tsan+0x19a58396) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)

  Previous write of size 8 at 0x7b50001b07a0 by thread T192:
    0 std::__1::basic_string<>::__set_long_cap[abi:v15000](unsigned long) build_docker/./contrib/llvm-project/libcxx/include/string:1592:37 (clickhouse-tsan+0x238809e2) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    1 std::__1::basic_string<>::__grow_by_and_replace(unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, unsigned long, char const*) build_docker/./contrib/llvm-project/libcxx/include/string:2367:5 (clickhouse-tsan+0x238809e2)
    2 std::__1::basic_string<>::append(char const*, unsigned long) build_docker/./contrib/llvm-project/libcxx/include/string:2692:9 (clickhouse-tsan+0x238809e2)
    3 std::__1::basic_string<>::append(char const*) build_docker/./contrib/llvm-project/libcxx/include/string:2849:12 (clickhouse-tsan+0x1faa2a58) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    4 Poco::Exception::extendedMessage() build_docker/./base/poco/Foundation/src/Exception.cpp:114:27 (clickhouse-tsan+0x1faa2a58)
    5 DB::Exception::addMessage() <null> (clickhouse-tsan+0x7e0d5f1) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    6 DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:70:23 (clickhouse-tsan+0x1b3f2fed) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    7 DB::ExecutionThreadContext::executeTask() build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:95:9 (clickhouse-tsan+0x1b3f2fed)
    8 DB::PipelineExecutor::executeStepImpl(unsigned long, std::__1::atomic<bool>*) build_docker/./src/Processors/Executors/PipelineExecutor.cpp:273:26 (clickhouse-tsan+0x1b3e5a10) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    9 DB::PipelineExecutor::executeSingleThread(unsigned long) build_docker/./src/Processors/Executors/PipelineExecutor.cpp:239:5 (clickhouse-tsan+0x1b3e4bb8) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    10 DB::PipelineExecutor::executeImpl(unsigned long, bool) build_docker/./src/Processors/Executors/PipelineExecutor.cpp:399:9 (clickhouse-tsan+0x1b3e4bb8)
    11 DB::PipelineExecutor::execute(unsigned long, bool) build_docker/./src/Processors/Executors/PipelineExecutor.cpp:111:9 (clickhouse-tsan+0x1b3e48be) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    12 DB::CompletedPipelineExecutor::execute() build_docker/./src/Processors/Executors/CompletedPipelineExecutor.cpp:110:18 (clickhouse-tsan+0x1b3e31c8) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    13 DB::FutureSetFromSubquery::buildOrderedSetInplace(std::__1::shared_ptr<DB::Context const> const&) build_docker/./src/Interpreters/PreparedSets.cpp:202:14 (clickhouse-tsan+0x19a58396) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)

  Location is heap block of size 480 at 0x7b50001b0600 allocated by thread T193:
    0 posix_memalign <null> (clickhouse-tsan+0x7d6ac85) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    1 std::__1::__libcpp_aligned_alloc[abi:v15000](unsigned long, unsigned long) build_docker/./contrib/llvm-project/libcxxabi/../libcxx/include/new:328:9 (clickhouse-tsan+0x238b5a05) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    2 __cxxabiv1::__aligned_malloc_with_fallback(unsigned long) build_docker/./contrib/llvm-project/libcxxabi/src/fallback_malloc.cpp:215:20 (clickhouse-tsan+0x238b5a05)
    3 __cxa_allocate_exception build_docker/./contrib/llvm-project/libcxxabi/src/cxa_exception.cpp:190:17 (clickhouse-tsan+0x238b31e5) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    4 std::exception_ptr std::make_exception_ptr[abi:v15000]<DB::Exception>(DB::Exception) build_docker/./contrib/llvm-project/libcxx/include/exception:206:9 (clickhouse-tsan+0x144bfa8a) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    5 DB::CreatingSetsTransform::~CreatingSetsTransform() build_docker/./src/Processors/Transforms/CreatingSetsTransform.cpp:30:45 (clickhouse-tsan+0x1b7603c3) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
    ...
    35 DB::FutureSetFromSubquery::buildOrderedSetInplace(std::__1::shared_ptr<DB::Context const> const&) build_docker/./src/Interpreters/PreparedSets.cpp:211:1 (clickhouse-tsan+0x19a58608) (BuildId: f978f51991d62a36512b8fc6952335a82292896f)
 2023-10-08 17:16:39 +02:00
Azat Khuzhin
099665478d Fix incorrect merging of Nested for SELECT FINAL FROM SummingMergeTree 
The problem was the order of the columns, in case of SELECT FINAL it got
"counters_Map.count", "counters_Map.id"

But in case of OPTIMIZE FINAL it got "counters_Map.id",
"counters_Map.count" correctly.

Note, that this bugs exists there from the very recent versions, I've
checked 19.x and it was there.

P.S. there is a workaround for this problem, if you will use one of the
following patterns for key columns:
- *ID
- *Key
- *Type

That way it will be explicitly matched as key and everything will work.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-10-08 07:32:47 +02:00
Nikolai Kochetov
3bd71c937c Merge branch 'master' into fix-read-in-order-with-array-join 2023-10-04 09:17:09 +00:00
Nikolai Kochetov
d944b59902 Merge branch 'master' into planner-prepare-filters-for-analysis-2 2023-10-03 14:28:16 +00:00
vdimir
3a9abde35d
Merge pull request #54514 from ClickHouse/vdimir/allow_experimental_partial_result 
Add setting allow_experimental_partial_result
 2023-09-29 10:32:57 +02:00
Alexander Gololobov
d81d5dd75c Cleanup in case of exception in ctor 2023-09-28 16:28:35 +02:00
Nikita Taranov
0e506b618e
impl (#54934) 2023-09-28 14:12:19 +02:00
vdimir
a134569bc3
fix MergeSortingPartialResultTransform::getRealProcessorSnapshot 2023-09-28 09:41:01 +00:00
vdimir
4e86eec7d1
fix Pipe::unitePipes with partial results 2023-09-28 09:41:00 +00:00
vdimir
3f3feea0b7
Add setting allow_experimental_partial_result 2023-09-28 09:40:56 +00:00
robot-clickhouse-ci-1
7a825c1417
Merge pull request #54427 from Avogar/json-object-as-tuple-inference 
Add new features to schema inference for JSON formats
 2023-09-27 20:12:45 +02:00
Robert Schulze
2d750e9e2f
Merge remote-tracking branch 'rschu1ze/master' into clang-tidy-reenable-checks 2023-09-27 12:15:29 +00:00
Antonio Andelic
580aa0031c
Merge pull request #55013 from ClickHouse/another-promise-fix 
Set exception for promise in `CreatingSetsTransform` in more cases
 2023-09-27 09:01:42 +02:00
Michael Kolupaev
23bc286280
Print more information about one logical error in MergeTreeDataWriter (#55001) 2023-09-26 13:23:26 -07:00
Han Fei
4e7745f757 Merge branch 'master' into hanfei/statistic 2023-09-26 21:43:53 +02:00
Robert Schulze
cde10fe7b5
Merge remote-tracking branch 'rschu1ze/master' into clang-tidy-reenable-checks 2023-09-26 18:59:41 +00:00
Kruglov Pavel
bea80ab5b7
Merge branch 'master' into json-object-as-tuple-inference 2023-09-26 15:23:08 +02:00
Kruglov Pavel
69a17bbef6
Merge pull request #52853 from Avogar/http-valid-json-on-exception 
Output valid JSON/XML on excetpion during HTTP query execution
 2023-09-26 14:25:55 +02:00
Robert Schulze
9fff447716
Re-enable clang-tidy checks 2023-09-26 09:34:12 +00:00
Antonio Andelic
098efb261e Throw proper exception if promise was not set 2023-09-26 09:03:38 +00:00
avogar
9e75825515 Merge branch 'master' of github.com:ClickHouse/ClickHouse into json-object-as-tuple-inference 2023-09-25 17:24:36 +00:00
avogar
42ca897f2d Better schema inference for JSON formats 2023-09-25 15:42:59 +00:00
Dmitry Novik
d4acd9ec61 Set correct stream sorting 2023-09-25 15:34:35 +00:00
Alex Cheng
d5d3cf0e9d Merge remote-tracking branch 'origin/master' into refactor-MergeJoinTransform 2023-09-25 19:41:35 +08:00
Alex Cheng
c7a88c8946 refactor MergeJoinTransform code. 2023-09-25 18:04:25 +08:00
robot-ch-test-poll4
9d61ed4665
Merge pull request #54664 from bigo-sg/arrow_join_max_block_size 
Respect max_block_size for array join to avoid possible OOM
 2023-09-23 21:30:51 +02:00
robot-ch-test-poll4
ba6f0431a5
Merge pull request #54933 from ClibMouse/feature/big-endian-bson-each-row 
Provide support for BSON on BE
 2023-09-23 03:00:27 +02:00
robot-clickhouse-ci-2
d98234dc9d
Merge pull request #54803 from Avogar/ephemeral-columns-from-files 
Forbid special columns for file/s3/url/... storages, fix insert into ephemeral columns from files
 2023-09-22 23:24:42 +02:00
kothiga
3e57b007a8
Use LE version of unalignedStore. 2023-09-22 12:25:17 -07:00
robot-clickhouse-ci-1
9a3d1d6296
Merge pull request #54920 from ClickHouse/set-promise-on-exception 
Set exception for promise in `CreatingSetsTransform`
 2023-09-22 20:45:09 +02:00
kothiga
80d511093b
Provide support for BSON on BE 2023-09-22 09:21:21 -07:00
Kruglov Pavel
4d675dbad4
Merge pull request #54825 from azat/fix-virtual-columns-filtering 
Fix filtering parts with indexHint for non analyzer (resubmit)
 2023-09-22 18:20:16 +02:00
Dmitry Novik
3b0579b339 Fix splitting into buckets 2023-09-22 14:44:40 +00:00
Antonio Andelic
a691dea6cf Set exception for promise 2023-09-22 13:13:54 +00:00
Robert Schulze
9a91f97ac4
Merge pull request #53831 from rschu1ze/clang-17 
Switch CI to Clang 17
 2023-09-22 12:28:35 +02:00
Igor Nikonov
b1cc698477
Merge pull request #54564 from vitlibar/fix-sorting-of-union-of-sorted 
Fix sorting of UNION ALL of already sorted results
 2023-09-21 22:49:53 +02:00
Robert Schulze
877e4f3aab
Merge remote-tracking branch 'rschu1ze/master' into clang-17 2023-09-21 20:21:12 +00:00
Michael Kolupaev
9af9b4a085
Enable connection pooling for s3 table function (#54812) 
Enable connection pooling for s3 table function
 2023-09-21 09:27:20 -07:00
Robert Schulze
5209bd2d51
Merge remote-tracking branch 'rschu1ze/master' into clang-17 2023-09-21 14:45:55 +00:00
Robert Schulze
f5137dd0b4
More clang-tidy fixes 2023-09-21 14:40:57 +00:00
avogar
3e08800cb5 Forbid special columns for file/s3/url/... storages, fix insert into ephemeral columns from files 2023-09-20 16:25:55 +00:00
Dmitry Novik
bb6952317c Merge remote-tracking branch 'origin/master' into parallel-window 2023-09-20 15:38:42 +00:00
Igor Nikonov
2d446fc1ca Merge remote-tracking branch 'origin/master' into parallel-replicas-not-enough-replicas 2023-09-20 13:40:08 +00:00
Kruglov Pavel
49ee14f701
Merge pull request #54809 from ClickHouse/pqmeta 
Prevent ParquetMetadata reading 40 MB from each file unnecessarily
 2023-09-20 12:53:22 +02:00
SmitaRKulkarni
135df6870c
Merge pull request #54706 from ClickHouse/replacing_merge_tree_with_clean_deleted_rows_disabled 
Fix ReplacingMergeTree with vertical merge and cleanup
 2023-09-20 12:06:55 +02:00
SmitaRKulkarni
d8adf05de2
Added a new column _block_number (#47532) 
Added a new virtual column _block_number which is persisted on merges when allow_experimental_block_number_column is enabled
 2023-09-20 11:31:12 +02:00
Azat Khuzhin
d9a634eb0f Fix filtering parts with indexHint for non analyzer 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
(cherry picked from commit ffa82e9297)
 2023-09-20 11:29:35 +02:00
Michael Kolupaev
c856ec4087 Prevent ParquetMetadata reading 40 MB from each file unnecessarily 2023-09-19 21:58:50 +00:00
Igor Nikonov
e28f4e819c
Merge branch 'master' into parallel-replicas-not-enough-replicas 2023-09-19 23:05:24 +02:00
Azat Khuzhin
c439c4bca2
Revert "Fix filtering parts with indexHint for non analyzer" 2023-09-19 21:39:21 +02:00
Han Fei
eb86c113e5
Merge pull request #54184 from priera/fix_mv_insertion_when_duplicated_parts 
Inserting only non-duplicate chunks in MV
 2023-09-19 20:41:45 +02:00
robot-ch-test-poll4
a3dcce21b3
Merge pull request #54405 from Avogar/json-or-jsonl 
Parse data in JSON format as JSONEachRow if failed to parse metadata
 2023-09-19 20:15:09 +02:00
Kruglov Pavel
d555fdffa5
Merge pull request #54449 from azat/parts-prune-indexHint 
Fix filtering parts with indexHint for non analyzer
 2023-09-19 19:15:41 +02:00
Han Fei
17243b84ab Merge branch 'master' into hanfei/statistic 2023-09-19 17:09:17 +02:00
Smita Kulkarni
e2887ad27b Merge branch 'master' into replacing_merge_tree_with_clean_deleted_rows_disabled 2023-09-19 15:58:41 +02:00
avogar
f974970c3c Apply suggestion 2023-09-19 11:53:40 +00:00
avogar
5bd2e9f610 Fix tests 2023-09-19 11:53:40 +00:00
avogar
8c29408f5e Parse data in JSON format as JSONEachRow if failed to parse metadata 2023-09-19 11:53:40 +00:00
Kruglov Pavel
e163670357
Merge branch 'master' into http-valid-json-on-exception 2023-09-19 13:42:53 +02:00
Kruglov Pavel
3c83e43351
Remove debug logging 2023-09-19 13:38:43 +02:00
priera
b92e37d2df Fixing PR comments 2023-09-19 12:15:01 +02:00
priera
663ce74609 fixed clang-tidy error 2023-09-19 12:15:00 +02:00
priera
b30e33580b fix test 2023-09-19 12:15:00 +02:00
priera
6d27ea2742 fixed style 2023-09-19 12:15:00 +02:00
priera
94d737264c inserting non-duplicate chunks 2023-09-19 12:15:00 +02:00
Smita Kulkarni
142a4631f1 Updated to clear current_row_sources when cleanedup rows are added 2023-09-19 11:22:10 +02:00
Igor Nikonov
047d214436 Merge remote-tracking branch 'origin/master' into parallel-replicas-not-enough-replicas 2023-09-18 15:29:56 +00:00
Igor Nikonov
e1019ba3c4 Disabling parallel replicas per shard will be done separately 2023-09-18 15:27:55 +00:00
Kruglov Pavel
9c888ea42b
Merge pull request #53549 from Avogar/group-by-constant-keys 
Optimize group by constant keys
 2023-09-18 12:12:40 +02:00
taiyang-li
39ca44b51a Respect max_block_size for array join to avoid possible OOM 2023-09-18 11:04:38 +08:00
Robert Schulze
f5e8028bb1
Merge pull request #54642 from rschu1ze/broken-re2st 
Remove broken lockless variant of re2
 2023-09-17 15:30:57 +02:00
Amos Bird
0518b64b58
Fix nullable primary key in final (#54164) 
* Fix nullable primary key in final

* Real fix

* Address reviews
 2023-09-15 22:44:13 +02:00
Smita Kulkarni
8ef910e1fa Updated to check if cleanup is enabled before adding to cleanedup_rows_count 2023-09-15 22:34:09 +02:00
Kruglov Pavel
2075f9c667
Merge branch 'master' into group-by-constant-keys 2023-09-15 15:10:08 +02:00
avogar
35d975bfea Add comment in ParallelInputFormat, remove unneded include 2023-09-15 13:07:04 +00:00
Kruglov Pavel
dbd24b240c
Merge branch 'master' into http-valid-json-on-exception 2023-09-15 14:55:31 +02:00
Sema Checherinda
45cf793544
Merge pull request #53919 from canhld94/optimize_all_lonely_parts 
Do not implicitly read pk and version columns in lonely parts if unnecessary
 2023-09-15 13:11:00 +02:00
Robert Schulze
7b378dbad3
Remove broken lockless variant of re2 2023-09-14 16:40:42 +00:00
Vitaly Baranov
9a0e1ef592 Fix sorting of UNION ALL of already sorted results. 2023-09-14 15:04:37 +02:00
Robert Schulze
a9ae813db0
Merge pull request #54115 from slvrtrn/simplified-prepared-statements-for-mysql 
Implement the MySQL binary protocol implementation for initial support of Tableau Online
 2023-09-14 12:27:11 +02:00
slvrtrn
c0961d9378 Merge remote-tracking branch 'origin' into simplified-prepared-statements-for-mysql 2023-09-13 19:33:11 +02:00
Sema Checherinda
8a9b544a97
Merge branch 'master' into optimize_all_lonely_parts 2023-09-13 16:07:19 +02:00
Arthur Passos
da8caeffd2 Merge branch 'master' into arrow_parquet_account_for_monotonically_increasing_offsets_across_batches 2023-09-12 17:50:36 -03:00
Igor Nikonov
7b3f32b95a
Merge pull request #54520 from ClickHouse/pr-cleanup 
Parallel replicas: cleanup unused params
 2023-09-12 19:48:18 +02:00