Commit Graph

14788 Commits

Author SHA1 Message Date
Igor Nikonov
823b62a55c
Merge pull request #56162 from ClickHouse/part-check-scheduling-monotonic-time
Use monotonic clock for part check scheduling
2023-11-01 23:15:27 +01:00
Igor Nikonov
447386edd9
Merge branch 'master' into fix-schedule-delayed-part-checks 2023-11-01 21:26:06 +01:00
Duc Canh Le
7d4c97e8f3
print broken part path before throw (#56181)
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
2023-11-01 19:11:23 +01:00
Azat Khuzhin
c25d6cd624
Rename directory monitor concept into background INSERT (#55978)
* Limit log frequence for "Skipping send data over distributed table" message

After SYSTEM STOP DISTRIBUTED SENDS it will constantly print this
message.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Rename directory monitor concept into async INSERT

Rename the following query settings (with preserving backward
compatiblity, by keeping old name as an alias):
- distributed_directory_monitor_sleep_time_ms -> distributed_async_insert_sleep_time_ms
- distributed_directory_monitor_max_sleep_time_ms -> distributed_async_insert_max_sleep_time_ms
- distributed_directory_monitor_batch -> distributed_async_insert_batch_inserts
- distributed_directory_monitor_split_batch_on_failure -> distributed_async_insert_split_batch_on_failure

Rename the following table settings (with preserving backward
compatiblity, by keeping old name as an alias):
- monitor_batch_inserts -> async_insert_batch
- monitor_split_batch_on_failure -> async_insert_split_batch_on_failure
- directory_monitor_sleep_time_ms -> async_insert_sleep_time_ms
- directory_monitor_max_sleep_time_ms -> async_insert_max_sleep_time_ms

And also update all the references:

    $ gg -e directory_monitor_ -e monitor_ tests docs | cut -d: -f1 | sort -u | xargs sed -e 's/distributed_directory_monitor_sleep_time_ms/distributed_async_insert_sleep_time_ms/g' -e 's/distributed_directory_monitor_max_sleep_time_ms/distributed_async_insert_max_sleep_time_ms/g' -e 's/distributed_directory_monitor_batch_inserts/distributed_async_insert_batch/g' -e 's/distributed_directory_monitor_split_batch_on_failure/distributed_async_insert_split_batch_on_failure/g' -e 's/monitor_batch_inserts/async_insert_batch/g' -e 's/monitor_split_batch_on_failure/async_insert_split_batch_on_failure/g' -e 's/monitor_sleep_time_ms/async_insert_sleep_time_ms/g' -e 's/monitor_max_sleep_time_ms/async_insert_max_sleep_time_ms/g' -i

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Rename async_insert for Distributed into background_insert

This will avoid amigibuity between general async INSERT's and INSERT
into Distributed, which are indeed background, so new term express it
even better.

Mostly done with:

    $ git di HEAD^ --name-only | xargs sed -i -e 's/distributed_async_insert/distributed_background_insert/g' -e 's/async_insert_batch/background_insert_batch/g' -e 's/async_insert_split_batch_on_failure/background_insert_split_batch_on_failure/g' -e 's/async_insert_sleep_time_ms/background_insert_sleep_time_ms/g' -e 's/async_insert_max_sleep_time_ms/background_insert_max_sleep_time_ms/g'

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

* Mark 02417_opentelemetry_insert_on_distributed_table as long

CI: https://s3.amazonaws.com/clickhouse-test-reports/55978/7a6abb03a0b507e29e999cb7e04f246a119c6f28/stateless_tests_flaky_check__asan_.html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>

---------

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-11-01 15:09:39 +01:00
Alexander Tokmakov
65f3cf81dc
Merge pull request #55987 from ClickHouse/fix_lost_part_count
Avoid counting lost parts twice
2023-11-01 15:07:27 +01:00
Kruglov Pavel
5533258091
Merge pull request #55477 from Avogar/streaming-errors
Allow to save unparsed records and errors in RabbitMQ, NATS and FileLog engines
2023-11-01 14:35:35 +01:00
vdimir
2ca1a14ac4
Merge pull request #55741 from ClickHouse/vdimir/fix_analyzer_materialized_distributed_join
Analyzer: fix block stucture mismatch in matview with engine distributed
2023-11-01 14:16:33 +01:00
jsc0218
f1c7665feb
Merge pull request #56121 from lithiumlabcompany/fix-auto-configuration-queue-for-librdkafka
Correctly set queued.min.messages
2023-10-31 19:59:04 -04:00
Michael Kolupaev
9c79c165b9
Re-fix 'Block structure mismatch' on concurrent ALTER and INSERTs in Buffer table (#56140)
* Fix 'Block structure mismatch' on concurrent ALTER and INSERTs in Buffer table

* Fix the test
2023-10-31 13:41:54 -07:00
Alexander Tokmakov
f39f0c6a74
Merge pull request #56083 from ClickHouse/fix_drop_range_commit_existing_part_race
Fix race condition between DROP_RANGE and committing existing block
2023-10-31 14:24:53 +01:00
Alexander Tokmakov
4d437ca462
Merge pull request #56137 from ClickHouse/fix_unexpected_parts_handling
Do not try to activate covered parts when handilng unexpected parts
2023-10-31 14:00:47 +01:00
Igor Nikonov
f8a7bdb8f0 Use monotonic time for part checks scheduling 2023-10-31 11:21:20 +00:00
Alexander Tokmakov
373d8bbb20 do not try to activate covered parts 2023-10-30 20:17:10 +01:00
Igor Nikonov
0958d0b168 Fix style 2023-10-30 16:35:00 +00:00
Nikolai Kochetov
cc3c038394 Fixing test. 2023-10-30 16:29:22 +00:00
Alexander Tokmakov
ae8d7a557b
Update ReplicatedMergeTreeSink.cpp 2023-10-30 15:43:48 +01:00
Alexander Tokmakov
073a6a6f0e
Update src/Storages/StorageReplicatedMergeTree.cpp
Co-authored-by: Raúl Marín <git@rmr.ninja>
2023-10-30 15:33:35 +01:00
Alexander Tokmakov
6635b84a46 fixes 2023-10-30 15:20:04 +01:00
Igor Nikonov
f2acc366f7 Fix: schedule delayed part checks correctly 2023-10-30 12:57:43 +00:00
Nikolai Kochetov
554ceb4e1d Merge branch 'master' into planner-prepare-filters-for-analysis-2 2023-10-30 11:56:30 +01:00
Stas Morozov
5a922014f7
Avoid problem with configuration queued.min.messages automatically for Kafka storage 2023-10-30 13:53:52 +03:00
Kruglov Pavel
e22cfa3d5b
Remove unneeded changge 2023-10-30 11:35:44 +01:00
Kruglov Pavel
53b6a455fc
Remove unused field from .h file 2023-10-30 11:35:09 +01:00
Kruglov Pavel
ba736beaf2
Remove unused field 2023-10-30 11:34:42 +01:00
Alexey Milovidov
3631e476eb
Merge pull request #54947 from amosbird/minmax-combinator
Introduce -ArgMin/-ArgMax combinators.
2023-10-30 03:35:28 +01:00
Alexey Milovidov
64b6e68a50
Merge pull request #55683 from amosbird/issue-55653
Reuse granule during skip index reading
2023-10-30 00:51:51 +01:00
Alexey Milovidov
9aff0a8872 Revert "Fix 'Block structure mismatch' on concurrent ALTER and INSERTs in Buffer table (#55995)"
This reverts commit b65c498016.
2023-10-29 04:34:04 +03:00
Alexey Milovidov
1b085b1161
Merge pull request #55900 from ClickHouse/make-test-better
Make test `system_zookeeper_connection` better
2023-10-28 15:01:36 +02:00
Michael Kolupaev
b65c498016
Fix 'Block structure mismatch' on concurrent ALTER and INSERTs in Buffer table (#55995) 2023-10-27 10:06:43 -07:00
Alexander Tokmakov
e07bb123d3 fix race condition between DROP_RANGE and committing existing block 2023-10-27 17:50:45 +02:00
Alexander Tokmakov
db5e9f1d1c beter check for duplicate lost parts 2023-10-27 17:33:30 +02:00
Alexander Tokmakov
9db5f4396c Merge branch 'master' into fix_lost_part_count 2023-10-26 17:40:37 +02:00
Andrey Zvonov
5737369d0b
Merge branch 'ClickHouse:master' into zvonand-datetime-ranges 2023-10-26 11:42:13 +02:00
vdimir
12890eb70c
Merge pull request #55806 from canhld94/rocksdb_trivial_count
Support trivial count for rocksdb and storage join
2023-10-26 11:22:38 +02:00
vdimir
1d5f75fd71
Merge pull request #53404 from ClickHouse/vdimir/check_table_improvements2
Improve CHECK TABLE system query, support checking part
2023-10-26 10:57:44 +02:00
zvonand
0766c73aab Rename date_time_overflow_mode -> date_time_overflow_behavior, moved it to format settings 2023-10-25 23:11:13 +02:00
Alexander Tokmakov
3c0b8aeb6a avoid counting lost parts twice 2023-10-25 19:29:40 +02:00
Alexander Tokmakov
c0482cb537
Merge pull request #55944 from azat/rmt-retriable-exception-log
Do not write retriable errors for Replicated mutate/merge into error log
2023-10-25 18:25:59 +02:00
alesapin
0d68a52c06
Merge pull request #50181 from ClickHouse/repro_49887
Make exception about `ALTER TABLE ... DROP COLUMN|INDEX|PROJECTION` more clear
2023-10-25 15:57:17 +02:00
李扬
465962df7f
Support orc filter push down (file + stripe + rowgroup level) (#55330)
* support orc filter push down

* update orc lib version

* replace setqueryinfo with setkeycondition

* fix issue https://github.com/ClickHouse/ClickHouse/issues/53536

* refactor source with key condition

* fix building error

* remove std::cout

* update orc

* update orc version

* fix bugs

* improve code

* upgrade orc lib

* fix code style

* change as requested

* add performance tests for orc filter push down

* add performance tests for orc filter push down

* fix all bugs

* fix default as null issue

* add uts for null as default issues

* upgrade orc lib

* fix failed orc lib uts and fix typo

* fix failed uts

* fix failed uts

* fix ast fuzzer tests

* fix bug of uint64 overflow in https://s3.amazonaws.com/clickhouse-test-reports/55330/de22fdcaea2e12c96f300e95f59beba84401712d/fuzzer_astfuzzerubsan/report.html

* fix asan fatal caused by reused column vector batch in native orc input format. refer to https://s3.amazonaws.com/clickhouse-test-reports/55330/be39d23af2d7e27f5ec7f168947cf75aeaabf674/stateless_tests__asan__[4_4].htm

* fix wrong performance tests

* disable 02892_orc_filter_pushdown on aarch64. https://s3.amazonaws.com/clickhouse-test-reports/55330/be39d23af2d7e27f5ec7f168947cf75aeaabf674/stateless_tests__aarch64_.html

* add some comments

* add some comments

* inline range::equals and range::less

* fix data race of key condition

* trigger ci
2023-10-24 12:08:17 -07:00
Kruglov Pavel
cb63b07e89
Merge pull request #55912 from wxybear/fix/storage_null_supports_subcolumns
fix: StorageNull supports subcolumns
2023-10-24 14:03:46 +02:00
Alexander Gololobov
04b82d6463
Merge pull request #55935 from ClickHouse/set_has_lwd
Set storage.has_lightweight_delete_parts flag when a part has been loaded
2023-10-24 14:01:11 +02:00
Azat Khuzhin
da408df4a7 Proper ignoring of retriable errors during merge/mutate
The exception cannot be simply suppressed, since sanity checks in the
WriteBuffer dtor relies on the std::uncaught_exceptions(), and if the
exception will be suppressed, then if the buffer was not finalized, it
may abort (even though it is only in debug/sanitizers build).

So instead, IExecutableTask::printExecutionException() had been
introduced to distinguish when exception should be printed and when not.
2023-10-24 10:19:18 +02:00
Duc Canh Le
e3aa799ca2 disable approximate count by default + fix tests
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
2023-10-24 03:18:33 +00:00
Srikanth Chekuri
8876a92b42 Merge remote-tracking branch 'upstream/master' into issue-55023 2023-10-23 21:21:25 +05:30
Robert Schulze
0703c88381
Merge pull request #55915 from rschu1ze/sanity-check-rpnbuildertreenode
Add sanity check to RPNBuilderFunctionTreeNode
2023-10-23 16:09:56 +02:00
Robert Schulze
601276aa60
Update src/Storages/MergeTree/RPNBuilder.cpp
Co-authored-by: Igor Nikonov <954088+devcrafter@users.noreply.github.com>
2023-10-23 15:23:23 +02:00
Azat Khuzhin
38a4547c05 Do not write retriable errors for Replicated mutate/merge into error log
Fixes: e3f892f84a ("fix gtest with MemoryWriteBuffer, do not mute exception in ReplicatedMergeMutateTaskBase")
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-10-23 15:14:58 +02:00
vdimir
7583394c9c
Support CHECK TABLE ... PART ... query 2023-10-23 12:33:35 +00:00
Kruglov Pavel
32da07b4ee
Merge pull request #55695 from Avogar/restrict-iceberg
Fixes and improvements for Iceberg storage
2023-10-23 13:58:23 +02:00