Commit Graph

5367 Commits

Author SHA1 Message Date
Alexander Tokmakov
0ce1560e5d retry if table replica being restarted 2023-11-02 18:30:32 +01:00
Alexey Milovidov
1267246785
Merge pull request #55642 from ClickHouse/morton-utils
Indexing by space-filling curves
2023-11-02 12:21:33 +01:00
Alexey Milovidov
697f2591b9 Merge with master 2023-11-02 00:33:33 +01:00
Kruglov Pavel
aa7c1f63ab
Merge pull request #56172 from Avogar/fix-schema-cache-for-json
Fix schema cache for fallback JSON->JSONEachRow with changed settings
2023-11-01 20:24:34 +01:00
Kruglov Pavel
e6f00d5e1d
Merge pull request #56117 from bigo-sg/fixed_struct_field_prune
Improve parquet struct fields reading
2023-11-01 15:32:50 +01:00
Kruglov Pavel
bf77ce691c
Merge pull request #55982 from yariks5s/npy_input_format
New input format Npy
2023-11-01 14:26:22 +01:00
Nikolai Kochetov
4deaf7cefb
Merge pull request #56134 from yariks5s/force_optimize_projection_name
Implementing force_optimize_projection_name
2023-11-01 13:12:10 +01:00
lgbo-ustc
8334585eaf improve parquet struct field reading 2023-11-01 15:18:39 +08:00
yariks5s
6c4bf59021 fix suggestions and enhance tests 2023-10-31 18:10:55 +00:00
avogar
518e52473d Fix schema cache for fallback JSON->JSONEachRow with changed settings 2023-10-31 14:12:38 +00:00
Nikolai Kochetov
f748f12426
Merge pull request #51746 from ClickHouse/fix-read-in-order-with-array-join
Fix 'Cannot find column' in read-in-order optimization with ARRAY JOIN
2023-10-31 11:51:01 +01:00
Nikolai Kochetov
0523e6cbd0
Merge pull request #53282 from ClickHouse/planner-prepare-filters-for-analysis-2
Support skip_unused_shards in Analyzer
2023-10-31 11:33:59 +01:00
Alexey Milovidov
467b4d85e2
Revert "Fix output/input of Arrow dictionary column" 2023-10-31 09:28:09 +03:00
Kruglov Pavel
4c2a132d96
Merge pull request #55989 from Avogar/lc-as-arrow-dict-fix
Fix output/input of Arrow dictionary column
2023-10-30 20:47:49 +01:00
Kruglov Pavel
4effc676f9
Merge pull request #56046 from Avogar/cr-in-unquoted-csv-string
Allow unquoted strings with CR in CSV format
2023-10-30 20:46:20 +01:00
Nikolai Kochetov
84f6a243b7 Merge branch 'master' into fix-read-in-order-with-array-join 2023-10-30 16:35:31 +00:00
yariks5s
03236c48ed init 2023-10-30 16:21:50 +00:00
yariks5s
9a2d89e3e4 removed getSize() and enhanced docs 2023-10-30 12:42:19 +00:00
Nikolai Kochetov
554ceb4e1d Merge branch 'master' into planner-prepare-filters-for-analysis-2 2023-10-30 11:56:30 +01:00
Kruglov Pavel
c10a3b3838
Merge branch 'master' into lc-as-arrow-dict-fix 2023-10-30 11:20:57 +01:00
avogar
57bc4854c2 Fix 2023-10-30 10:17:49 +00:00
avogar
d1fcbc6e47 Fix fetching schema from schema registry in AvroConfluent 2023-10-30 10:17:48 +00:00
yariks5s
e14a7f066a fix typos 2023-10-28 01:46:59 +00:00
yariks5s
894724bfb3 suggested changes 2023-10-28 01:17:25 +00:00
yariks5s
23635352f1 fixed due to review 2023-10-27 15:43:03 +00:00
frinkr
18c50c11b3
Multithreading after window functions (#50771)
* feat: Preserve number of streams after evaluation the window functions to allow parallel stream processing

* fix style

* fix style

* fix style

* setting query_plan_preserve_num_streams_after_window_functions default true

* fix tests by SETTINGS query_plan_preserve_num_streams_after_window_functions=0

* fix test references

* Resize the streams after the last window function, to keep the order between WindowTransforms (and WindowTransform works on single stream anyway).

* feat: Preserve number of streams after evaluation the window functions to allow parallel stream processing

* fix style

* fix style

* fix style

* setting query_plan_preserve_num_streams_after_window_functions default true

* fix tests by SETTINGS query_plan_preserve_num_streams_after_window_functions=0

* fix test references

* Resize the streams after the last window function, to keep the order between WindowTransforms (and WindowTransform works on single stream anyway).

* add perf test

* perf: change the dataset from 50M to 5M

* rename query_plan_preserve_num_streams_after_window_functions -> query_plan_enable_multithreading_after_window_functions

* update test reference

* fix clang-tidy

---------

Co-authored-by: Nikita Taranov <nikita.taranov@clickhouse.com>
2023-10-27 12:36:28 +02:00
avogar
9d207bf027 Allow unquoted strings with CR in CSV format 2023-10-26 13:50:54 +00:00
zvonand
0766c73aab Rename date_time_overflow_mode -> date_time_overflow_behavior, moved it to format settings 2023-10-25 23:11:13 +02:00
zvonand
5b86e8c714 updated after review 2023-10-25 23:10:58 +02:00
zvonand
2f3695add8 Introduce setting for dt overflow exception
Added tests and docs
2023-10-25 23:10:24 +02:00
Alexey Milovidov
bb5a60dc19
Merge pull request #55893 from ClickHouse/revert-partial-result-2
Revert "Revert "Revert "Add settings for real-time updates during query execution"""
2023-10-25 22:20:28 +02:00
yariks5s
2ab1ae42c1 added docs and tests, style check 2023-10-25 10:37:05 +00:00
robot-ch-test-poll1
ef78889aa2
Merge pull request #55952 from ClickHouse/disable_apply_deleted_mask
Added a setting to allow reading rows marked as deleted
2023-10-25 01:14:07 +02:00
avogar
c080ba9d7e Fix output LowCardinality as Arrow dictionary 2023-10-24 19:49:04 +00:00
李扬
465962df7f
Support orc filter push down (file + stripe + rowgroup level) (#55330)
* support orc filter push down

* update orc lib version

* replace setqueryinfo with setkeycondition

* fix issue https://github.com/ClickHouse/ClickHouse/issues/53536

* refactor source with key condition

* fix building error

* remove std::cout

* update orc

* update orc version

* fix bugs

* improve code

* upgrade orc lib

* fix code style

* change as requested

* add performance tests for orc filter push down

* add performance tests for orc filter push down

* fix all bugs

* fix default as null issue

* add uts for null as default issues

* upgrade orc lib

* fix failed orc lib uts and fix typo

* fix failed uts

* fix failed uts

* fix ast fuzzer tests

* fix bug of uint64 overflow in https://s3.amazonaws.com/clickhouse-test-reports/55330/de22fdcaea2e12c96f300e95f59beba84401712d/fuzzer_astfuzzerubsan/report.html

* fix asan fatal caused by reused column vector batch in native orc input format. refer to https://s3.amazonaws.com/clickhouse-test-reports/55330/be39d23af2d7e27f5ec7f168947cf75aeaabf674/stateless_tests__asan__[4_4].htm

* fix wrong performance tests

* disable 02892_orc_filter_pushdown on aarch64. https://s3.amazonaws.com/clickhouse-test-reports/55330/be39d23af2d7e27f5ec7f168947cf75aeaabf674/stateless_tests__aarch64_.html

* add some comments

* add some comments

* inline range::equals and range::less

* fix data race of key condition

* trigger ci
2023-10-24 12:08:17 -07:00
yariks5s
4e09fb3e27 made_logic 2023-10-24 14:55:52 +00:00
Kruglov Pavel
9a56ec4d63
Merge pull request #55891 from Avogar/try-fix-orc
Try to fix possible segfault in Native ORC input format
2023-10-24 13:57:43 +02:00
Alexander Gololobov
959b8b64bd Added a setting to allow reading rows marked as deleted 2023-10-23 19:59:17 +02:00
Robert Schulze
bef5cea5a0
Merge remote-tracking branch 'rschu1ze/master' into string-addinterval 2023-10-23 14:08:18 +00:00
Kruglov Pavel
32da07b4ee
Merge pull request #55695 from Avogar/restrict-iceberg
Fixes and improvements for Iceberg storage
2023-10-23 13:58:23 +02:00
Robert Schulze
c2215cb5f0
Support string arguments 2023-10-22 09:28:45 +00:00
Alexey Milovidov
5217d64551 Remove garbage 2023-10-22 01:53:50 +02:00
Alexey Milovidov
1be4ff229b
Merge pull request #55895 from ClickHouse/fix-window-functions-with-sparse-columns
Fix window functions in case of sparse columns.
2023-10-22 00:02:34 +02:00
János Benjamin Antal
1b0cadebd0 Convert sparse columns to full for window functions 2023-10-21 11:46:25 +00:00
Alexey Milovidov
7ec4b99e94 Revert partial result 2023-10-21 03:14:22 +02:00
avogar
8cc0dc17eb Try to fix possible segfault in Native ORC input format 2023-10-20 18:50:48 +00:00
yariks5s
87f26f5132 dealt with 2dim arrays 2023-10-20 17:05:05 +00:00
Anton Popov
5819bcd07a
Support asynchronous inserts for native protocol (#54730)
* support async insert for native protocol

* use separate queue for async inserts via native protocol

* fix test

* better logging for async inserts and more tests

* disable mixed internal and external data in async inserts

* fix tests

* fix quota in async inserts

* disable async insert for secondary query of distributed
2023-10-20 18:39:48 +02:00
Raúl Marín
d08ce13107
Merge pull request #51692 from Algunenano/parallel_replicas_row_estimation
Automatic decision of nº of parallel replicas
2023-10-20 15:50:43 +02:00
Azat Khuzhin
d5c960ceb6 Fix data race in CreatingSetsTransform
The problem is that findOrPromiseToBuild() can also call
shared_future::get():

    (lldb) bt 10
    * thread 194, name = 'MergeMutate', stop reason = step over
      * frame 0: 0x0000564126046bd2 clickhouse`std::rethrow_exception(p=(__ptr_ = 0x00007b5000130c80)) at exception_pointer_cxxabi.ipp:68:39
        frame 1: 0x000056411c4c5ddd clickhouse`std::__1::__assoc_state<std::__1::shared_ptr<DB::Set>>::copy(this=<unavailable>) at future:701:9
        frame 2: 0x000056411c4c0e3f clickhouse`DB::PreparedSetsCache::findOrPromiseToBuild(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) [inlined] std::__1::shared_future<std::__1::shared_ptr<DB::Set>>::get[abi:v15000](this=0x00007b0c00154eb8) const at future:2243:46
        frame 3: 0x000056411c4c0e2e clickhouse`DB::PreparedSetsCache::findOrPromiseToBuild(this=0x00007b1c00077018, key="__set_10438532338982543705_15680061365857338647") at PreparedSets.cpp:349:116

Note, that my test did not find the problem because of #55261 had not
been merged yet.

Follow-up for: #55338
Fixes: #55279
v2: s/createDeprecated/createRuntime/

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-10-19 12:26:02 +02:00