thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-14 18:32:29 +00:00
Code Issues Packages Projects Releases Wiki Activity
131,497 Commits 9,328 Branches 1,783 Tags 2.1 GiB
5dc04a13a7
Commit Graph

15578 Commits

Author SHA1 Message Date
Kseniia Sumarokova
b3f22449f6
Merge pull request #57515 from ClickHouse/fix-materialized-pg-issue-with-incorrect-connection-options 
MaterializedPostgreSQL: fix issue #41922, add test for #41923
 2023-12-07 10:42:14 +01:00
Alexey Milovidov
b85214ca1a
Merge pull request #57546 from rschu1ze/revert-revert-support_alias_column_in_indices 
Revert "Revert "Support use alias column in indices""
 2023-12-07 01:22:03 +01:00
Alexey Milovidov
5fe1b8926c
Merge pull request #57595 from ClickHouse/23.12-prepare 
Update version after release
 2023-12-06 23:53:53 +01:00
Alexey Milovidov
c8fb574fc8 Update autogenerated version to 23.12.1.1 and contributors 2023-12-06 23:53:06 +01:00
Yakov Olkhovskiy
ac7210b9b3
Merge pull request #57461 from ClickHouse/fix-ephemeral-matview 
Fix: don't exclude ephemeral column when building pushing to view chain
 2023-12-06 16:46:54 -05:00
Alexey Milovidov
05bc8ef1e0
Merge pull request #57548 from ClickHouse/s3queue_production_ready 
`S3Queue` is production ready
 2023-12-06 22:21:08 +01:00
Alexey Milovidov
3723573f81 Revert "Merge pull request #56724 from canhld94/ch_replicated_column_mismatch" 
This reverts commit a9bb76378f, reversing
changes made to 588fd16518.
 2023-12-06 19:09:07 +01:00
kssenii
806061642a Support GENERATED and DEFAULT columns 2023-12-06 15:01:43 +01:00
vdimir
8f32293104
Merge pull request #57416 from ClickHouse/vdimir/fix_test_encrypted_disk 
Change type of s3_cache in test_encrypted_disk
 2023-12-06 14:37:40 +01:00
Yakov Olkhovskiy
8f00678abe fix style 2023-12-06 13:24:25 +00:00
Alexey Milovidov
f8de7f6bc2 Merge branch 'master' into s3queue_production_ready 2023-12-06 12:25:29 +01:00
Kseniia Sumarokova
e02883cd4a
Merge branch 'master' into fix-materialized-pg-issue-with-incorrect-connection-options 2023-12-06 12:16:04 +01:00
alesapin
e8b017d323
Merge pull request #57513 from ClickHouse/enable_async_block_ids_cache_by_default 
Enable async block ids cache by default
 2023-12-06 11:35:25 +01:00
Nikolai Kochetov
718b542f40
Merge pull request #57518 from ClickHouse/cleanup-around-distributed 
Simple cleanup in distributed (while dealing with parallel replicas)
 2023-12-06 10:22:17 +01:00
Yakov Olkhovskiy
f5fb169c65 add system.dropped_tables_parts table 2023-12-06 06:00:00 +00:00
Nikolay Degterinsky
ed618d3be1 Fix crash in clickhouse-local 2023-12-06 04:24:43 +00:00
Alexey Milovidov
76bbeaa676 S3Queue is production ready 2023-12-05 21:10:19 +01:00
Robert Schulze
8e4e37d8ed
Revert "Revert "Support use alias column in indices"" 
This reverts commit d0b591be2a.
 2023-12-05 18:22:32 +00:00
Alexey Milovidov
d0b591be2a
Revert "Support use alias column in indices" 2023-12-05 18:21:45 +03:00
Anton Popov
eb990d863d fix tests 2023-12-05 15:15:46 +00:00
Anton Popov
913ae1d267 Merge remote-tracking branch 'upstream/master' into HEAD 2023-12-05 14:47:32 +00:00
Nikita Mikhaylov
04d167c6d9 Better 2023-12-05 13:34:37 +01:00
Igor Nikonov
87a4c0bde4 Simple cleanup in distributed (while dealing with parallel replicas) 2023-12-05 12:25:02 +00:00
kssenii
0d313acd03 Fix 2023-12-05 13:22:35 +01:00
Alexander Sapin
93a16b7cd6 Fix comments 2023-12-05 12:39:00 +01:00
Alexander Sapin
2587f368a6 Enable async block ids cache by default 2023-12-05 12:37:22 +01:00
Anton Popov
5770def9af
Merge pull request #57491 from jkartseva/limit-table-projections 
limit merge tree projections
 2023-12-05 12:13:41 +01:00
Julia Kartseva
2ac104240d limit merge tree projections 
Fixes: https://github.com/ClickHouse/ClickHouse/issues/56427
 2023-12-05 06:44:39 +00:00
Alexey Milovidov
10d65a1ade
Merge pull request #55559 from azat/s3-fix-excessive-reads 
Add ability to disable checksums for S3 to avoid excessive input file read
 2023-12-05 06:34:21 +01:00
kssenii
4a28f10c3d Minor cache changes 2023-12-04 19:02:37 +01:00
Alexey Milovidov
02439eee58
Merge pull request #49351 from ClickHouse/async-loader-integration 
Asynchronous loading of tables
 2023-12-04 17:16:12 +01:00
Azat Khuzhin
7986fe619a Introduce DistributedAsyncInsertionFailures - event for async INSERT failures 
Useful for alerts

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-04 16:52:53 +01:00
Azat Khuzhin
604cec475a Fix possible distributed sends stuck due to "No such file or directory" 
In case of restoring from current_batch.txt it is possible that the some
file from the batch will not be exist, and the fix submitted in #49884
was not complete, since it will fail later in markAsSend() (due to it
tries to obtain file size there):

    2023.12.04 05:43:12.676658 [ 5006 ] {} <Error> dist.DirectoryMonitor.work4: std::exception. Code: 1001, type: std::__1::__fs::filesystem::filesystem_error, e.what() = filesystem error: in file_size: No such file or directory ["/work4/clickhouse/data/dist/shard8_all_replicas//150426396.bin"], Stack trace (when copying this message, always include the lines below):

    0. ./.build/./contrib/llvm-project/libcxx/include/exception:134: std::runtime_error::runtime_error(String const&) @ 0x00000000177e83f4 in /usr/lib/debug/usr/bin/clickhouse.debug
    1. ./.build/./contrib/llvm-project/libcxx/include/string:1499: std::system_error::system_error(std::error_code, String const&) @ 0x00000000177f0fd5 in /usr/lib/debug/usr/bin/clickhouse.debug
    2. ./.build/./contrib/llvm-project/libcxx/include/__filesystem/filesystem_error.h:42: std::__fs::filesystem::filesystem_error::filesystem_error[abi:v15000](String const&, std::__fs::filesystem::path const&, std::error_code) @ 0x000000000b844ca1 in /usr/lib/debug/usr/bin/clickhouse.debug
    3. ./.build/./contrib/llvm-project/libcxx/include/__filesystem/filesystem_error.h:90: void std::__fs::filesystem::__throw_filesystem_error[abi:v15000]<String&, std::__fs::filesystem::path const&, std::error_code const&>(String&, std::__fs::filesystem::path const&, std::error_code const&) @ 0x000000001778f953 in /usr/lib/debug/usr/bin/clickhouse.debug
    4. ./.build/./contrib/llvm-project/libcxx/src/filesystem/filesystem_common.h:0: std::__fs::filesystem::detail::(anonymous namespace)::ErrorHandler<unsigned long>::report(std::error_code const&) const @ 0x0000000017793ef7 in /usr/lib/debug/usr/bin/clickhouse.debug
    5. ./.build/./contrib/llvm-project/libcxx/src/filesystem/operations.cpp:0: std::__fs::filesystem::__file_size(std::__fs::filesystem::path const&, std::error_code*) @ 0x0000000017793e26 in /usr/lib/debug/usr/bin/clickhouse.debug
    6. ./.build/./src/Storages/Distributed/DistributedAsyncInsertDirectoryQueue.cpp:707: DB::DistributedAsyncInsertDirectoryQueue::markAsSend(String const&) @ 0x0000000011cd92c5 in /usr/lib/debug/usr/bin/clickhouse.debug
    7. ./.build/./contrib/llvm-project/libcxx/include/__iterator/wrap_iter.h💯 DB::DistributedAsyncInsertBatch::send() @ 0x0000000011cdd81c in /usr/lib/debug/usr/bin/clickhouse.debug
    8. ./.build/./src/Storages/Distributed/DistributedAsyncInsertDirectoryQueue.cpp:0: DB::DistributedAsyncInsertDirectoryQueue::processFilesWithBatching() @ 0x0000000011cd5054 in /usr/lib/debug/usr/bin/clickhouse.debug
    9. ./.build/./src/Storages/Distributed/DistributedAsyncInsertDirectoryQueue.cpp:417: DB::DistributedAsyncInsertDirectoryQueue::processFiles() @ 0x0000000011cd3440 in /usr/lib/debug/usr/bin/clickhouse.debug
    10. ./.build/./src/Storages/Distributed/DistributedAsyncInsertDirectoryQueue.cpp:0: DB::DistributedAsyncInsertDirectoryQueue::run() @ 0x0000000011cd3878 in /usr/lib/debug/usr/bin/clickhouse.debug
    11. ./.build/./contrib/llvm-project/libcxx/include/__functional/function.h:0: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x00000000103dbc34 in /usr/lib/debug/usr/bin/clickhouse.debug
    12. ./.build/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:701: DB::BackgroundSchedulePool::threadFunction() @ 0x00000000103de1b6 in /usr/lib/debug/usr/bin/clickhouse.debug
    13. ./.build/./src/Core/BackgroundSchedulePool.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000103de7d1 in /usr/lib/debug/usr/bin/clickhouse.debug
    14. ./.build/./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000b8c5502 in /usr/lib/debug/usr/bin/clickhouse.debug
    15. ./.build/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000b8c936e in /usr/lib/debug/usr/bin/clickhouse.debug
    16. ? @ 0x00007f1be8b30fd4 in ?
    17. ? @ 0x00007f1be8bb15bc in ?

And instead of ignoring errors, DistributedAsyncInsertBatch::valid() had
been added, that should be called when the files had been read from the
current_batch.txt, if it is not valid (some files from the batch did not
exist), then there is no sense in trying to send the same batch, so just
this file will be ignored, and files will be processed in a regular
order.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-04 16:52:53 +01:00
Azat Khuzhin
638d0102f8 Fix error_count in case of distributed_directory_monitor_max_sleep_time_ms>5min 
In this case the error counter will be decremented everytime.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-12-04 16:52:53 +01:00
vdimir
7e624ae911
Review fixes 2023-12-04 13:56:51 +00:00
vdimir
a4ae90de0d
Merge pull request #57275 from ClickHouse/vdimir/merge_task_tmp_data 
Background merges correctly use temporary data storage in the cache
 2023-12-04 14:52:20 +01:00
Anton Popov
8d30e22a09 fix lighweight delete with heavy delete 2023-12-04 13:00:50 +00:00
vdimir
b2e29d1416
Change type of s3_cache in test_encrypted_disk 2023-12-04 10:34:13 +00:00
robot-ch-test-poll1
d63b652dae
Merge pull request #57385 from vitlibar/fix-inconsistent-metadata-for-backup-2 
Stop using INCONSISTENT_METADATA_FOR_BACKUP that much
 2023-12-04 10:29:57 +01:00
Yakov Olkhovskiy
b601b66146 don't exclude ephemeral column when pushing to view 2023-12-04 06:56:33 +00:00
Vitaly Baranov
0457aba4a9 Show owner query ids in the message for the DEADLOCK_AVOIDED error. 2023-12-04 00:15:31 +01:00
Nikolay Degterinsky
2150308c23
Merge pull request #56948 from amosbird/fix-56932 
Prevent incompatible ALTER of projection columns
 2023-12-03 04:24:51 +01:00
Sema Checherinda
c61cd6aab2
Merge pull request #57335 from sichenzhao/increase_default_replicated_deduplication_window 
increase default replicated_deduplication_window from 100 to 1k
 2023-12-02 21:23:39 +01:00
Nikolay Degterinsky
a53e604429
Merge branch 'master' into fix-56932 2023-12-02 21:19:45 +01:00
Sema Checherinda
32e3b2657b
Merge pull request #57205 from CheSema/zero-locks-leak 
fix zero-copy locks leaking
 2023-12-02 15:54:12 +01:00
Sema Checherinda
a1c9501c07 fix 01158_zookeeper_log_long and 02442_auxiliary_zookeeper_endpoint 2023-12-02 04:57:33 +01:00
Anton Popov
7fbe7462b6 add mutation command to apply deleted mask 2023-12-01 19:12:05 +00:00
JackyWoo
8a68d51261
Add index to table system.numbers (#50909) 2023-12-01 19:59:25 +01:00
robot-ch-test-poll
1b49463bd2
Merge pull request #55841 from nickitat/optimize_reading3 
Optimize reading from cache
 2023-12-01 17:36:57 +01:00
Nikolay Degterinsky
ceed0f73d3
Merge pull request #57391 from evillique/better-partitioned-write-to-file 
Allow wildcards in directories for partitioned write with File storage
 2023-12-01 17:23:54 +01:00
Nikolai Kochetov
5430ac2c02
Merge pull request #57099 from ClickHouse/analyzer-fuzz-3 
Analyzer fuzzer 3 (aggregate_functions_null_for_empty for projections)
 2023-12-01 16:57:34 +01:00
Anton Popov
61c517bc02
Merge pull request #57352 from kitaisreal/mutations-reuse-source-part-granularity 
MergeTree mutations reuse source part index granularity
 2023-12-01 16:45:41 +01:00
Nikolai Kochetov
0b4131546a
Merge pull request #56813 from jsc0218/SystemTablesFilterEngine 
Able to Filter Engine When Scanning System Tables
 2023-12-01 16:02:27 +01:00
alesapin
fb98b212c5
Merge pull request #57311 from ClickHouse/make_alter_mat_view_non_experimental 
Make alter materialized view non experimental
 2023-12-01 12:18:43 +01:00
vdimir
fd373024f5
Update buffer re-reading in MergeTask.cpp 2023-12-01 10:11:08 +00:00
Sema Checherinda
82270627ed fix the relock 2023-12-01 02:12:35 +01:00
Vitaly Baranov
1bc1563e0e Stop using INCONSISTENT_METADATA_FOR_BACKUP that much. If possible prefer to continue scanning instead of stopping and starting the scanning for backup again. 2023-11-30 21:37:42 +01:00
Sema Checherinda
47daa4f235 remarks 2023-11-30 21:13:56 +01:00
Sema Checherinda
42f648f0d2 relock block when sessin expired 2023-11-30 21:13:56 +01:00
Sema Checherinda
e4becc01ba make stages commit 2023-11-30 21:13:56 +01:00
Sema Checherinda
7efa76f08d remove rename_part_to_temporary 2023-11-30 21:13:56 +01:00
Sema Checherinda
2cc31752b1 unlockSharedData in 1 more place, and do it in a best effort way 2023-11-30 21:13:56 +01:00
Sema Checherinda
ad6897ddbd make shellcheck happy 2023-11-30 21:13:55 +01:00
Sema Checherinda
ad7b51c3c8 fix zero locks leak 2023-11-30 21:13:55 +01:00
Sergey Kviatkevich
376d297da2 fix alter table tests 2023-11-30 21:42:13 +03:00
Sergey Kviatkevich
afae432e99 allow ALTER for TEMPORARY table 2023-11-30 21:42:12 +03:00
Nikolay Degterinsky
95b5b9eb35 Fix partitioned write for File storage 2023-11-30 16:18:12 +00:00
Kruglov Pavel
575ee1e665
Merge pull request #56868 from zvonand/zvonand-fileCluster 
Introduce `fileCluster` table function
 2023-11-30 13:49:53 +01:00
Antonio Andelic
9955f571ef
Merge pull request #57318 from mkmkme/mkmkme/remove-unused-vars 
Remove unused Strings from MergeTreeData
 2023-11-30 13:48:54 +01:00
vdimir
54fe2e1410
Fix clang tidy 2023-11-30 12:40:34 +00:00
alesapin
d8bdbf29c4 Merge branch 'master' into make_alter_mat_view_non_experimental 2023-11-30 11:14:25 +01:00
robot-ch-test-poll
07ca6236f6
Merge pull request #57372 from jkartseva/fuzz-json-verbose-exception 
verbose exception messages for StorageFuzzJSON
 2023-11-30 08:20:54 +01:00
zvonand
6730a29a0b upd 2023-11-30 01:28:25 +01:00
Julia Kartseva
ed65c26faf verbose exception messages for StorageFuzzJSON 
Follow-up to https://github.com/ClickHouse/ClickHouse/pull/56490#discussion_r1409556111
 2023-11-29 20:42:21 +00:00
Kruglov Pavel
c6fecfb1af
Merge pull request #56901 from KevinyhZou/Fix_allow_cr_end_of_csv_line 
Fix allow cr end of line for csv
 2023-11-29 20:57:58 +01:00
Nikolai Kochetov
a36c11b21e Disable transform_null_in as well just in case. 2023-11-29 17:30:22 +00:00
Nikolai Kochetov
fc02b88596 Merge branch 'master' into analyzer-fuzz-3 2023-11-29 17:19:56 +00:00
Nikolai Kochetov
dc43a36ee2 Fixing style 2023-11-29 16:40:18 +00:00
vdimir
b5babe1692
MergeTask uses temporary data storage 2023-11-29 16:18:32 +00:00
Nikolai Kochetov
390c27f7d8 Ignore aggregate_functions_null_for_empty for projections at insert. 2023-11-29 16:16:22 +00:00
zvonand
c329e382ab resolve conflicts 2023-11-29 16:03:07 +01:00
zvonand
a4d9fbdfc2 Fix indents, fix style, fix test 2023-11-29 15:55:24 +01:00
Kruglov Pavel
30148972ed
Merge pull request #57328 from Avogar/size-virtual-column-table-functions 
Add information about new _size virtual column in file/s3/url/hdfs/azure table functions
 2023-11-29 15:05:28 +01:00
Maksim Kita
6578fe5baa Updated implementation 2023-11-29 16:21:07 +03:00
Maksim Kita
f0e53fde64 MergeTree mutations reuse source part index granularity 2023-11-29 14:36:57 +03:00
Han Fei
4c0efb0262
Merge pull request #53240 from hanfei1991/hanfei/statistic 
use statistic to order prewhere conditions better
 2023-11-29 12:26:44 +01:00
Nikolai Kochetov
731a1bfcf8 Cleanup 2023-11-29 11:11:58 +00:00
vdimir
470a092e3d
Merge pull request #51745 from ClickHouse/vdimir/setting_join_algorithm_order 
Setting JoinAlgorithm respect specified order
 2023-11-29 11:24:39 +01:00
Raúl Marín
b102145a4f
Merge pull request #57301 from Algunenano/fix_reports 
Fix code reports
 2023-11-29 11:11:33 +01:00
Chen Lixiang
88920f08f1 add tests and fix whitespace issue 2023-11-29 17:26:11 +08:00
Robert Schulze
bc3da5032e
Merge pull request #57330 from rschu1ze/fix-no-lib-build 
Follow-up to #56490: Fix build with `cmake -DENABLE_LIBRARIES=0`
 2023-11-29 09:42:12 +01:00
Antonio Andelic
f8811351fa
Merge pull request #57267 from ClickHouse/fix-readonly-replica 
Fix `ReadonlyReplica` metric for all cases
 2023-11-29 08:32:25 +01:00
Sichen Zhao
1b8b1c0a63 increase default replicated_deduplication_window from 100 to 1k 2023-11-28 14:24:06 -08:00
Robert Schulze
5f5e8633c7
Merge pull request #57220 from ucasfl/alias_column_in_indices 
Support use alias column in indices
 2023-11-28 21:31:34 +01:00
Robert Schulze
7d26990752
Follow-up to #56490: Fix build with -DENABLE_LIBRARIES=0 2023-11-28 19:59:43 +00:00
vdimir
b748405976
Intorduce *List definition for muli enum settings 2023-11-28 19:09:02 +00:00
vdimir
a6eb2ad930
Fix JoinStep::describeJoinActions 2023-11-28 19:09:01 +00:00
Nikolai Kochetov
4d906d6e3c Cleanup 2023-11-28 18:35:19 +00:00
avogar
4d9a1b50f9 Add information about new _size virtual column in file/s3/url/hdfs/azure table functions 2023-11-28 18:15:07 +00:00
Nikolai Kochetov
84194f10f2 Fixing tests. 2023-11-28 18:04:30 +00:00
Nikolai Kochetov
08a7575984 Re-implement filtering a bit. 2023-11-28 16:17:35 +00:00
Han Fei
e4421e2808 remove wrong code 2023-11-28 16:46:55 +01:00
Nikita Taranov
52f644c0df Merge branch 'master' into optimize_reading3 2023-11-28 16:36:38 +01:00
Han Fei
6733438adc Merge branch 'master' into hanfei/statistic 2023-11-28 16:35:00 +01:00
Han Fei
8e84c734bf rename some code 2023-11-28 16:32:47 +01:00
zvonand
c306d21b54 merge master + resolve conflicts 2023-11-28 15:51:21 +01:00
Mikhail Koviazin
daaa575106
Remove unused Strings from MergeTreeData 
Compiler does not emit a warning when there is an unused variable of
a type with a non-trivial constructor. std::string is one of them.
Hence, these two variables were overlooked. This commit removes them
 2023-11-28 16:33:44 +02:00
Alexander Tokmakov
e40c71a74c
Merge pull request #56533 from ClickHouse/rmt_check_shutdown_flags_in_retry_loops 
ReplicatedMergeTree: check shutdown flags in retry loops
 2023-11-28 15:24:40 +01:00
Alexander Sapin
2cd9f5f3cc Fix known issue 2023-11-28 15:13:12 +01:00
Alexander Sapin
e6855cef4d Followup 2023-11-28 14:01:31 +01:00
Alexander Sapin
b5404f4a9f Followup 2023-11-28 13:34:05 +01:00
Alexander Sapin
e1aa62d9cd Make alter materialized view non expiremental and deprecate allow_experimental_alter_materialized_view_structure obsolete 2023-11-28 13:17:55 +01:00
Antonio Andelic
636ac3f349 Fix assert 2023-11-28 12:07:05 +00:00
vdimir
af81cb8252
Setting JoinAlgorithm respect specified order 2023-11-28 11:01:35 +00:00
Antonio Andelic
9dd4783675 Merge branch 'master' into fix-readonly-replica 2023-11-28 10:43:29 +00:00
Robert Schulze
d459fa1fc6
Merge remote-tracking branch 'ClickHouse/master' into alias_column_in_indices 2023-11-28 10:15:25 +00:00
Raúl Marín
ff0df49e2b Fix comment in MergeTreeDataPartWide::checkConsistency 2023-11-28 10:09:03 +00:00
Nikolai Kochetov
7e1f625179 Revert system tables to master. 2023-11-28 10:07:36 +00:00
Julia Kartseva
57a4492275
Merge pull request #56490 from jkartseva/fuzz_json 
Fuzz JSON table function
 2023-11-27 21:47:53 -08:00
Ubuntu
a85cf758b0 Fuzz JSON table function 
Create a table function `fuzzJSON`

An example query:
```
CREATE NAMED COLLECTION json_fuzzer AS json_str={};

SELECT *
FROM fuzzJSON(json_fuzzer, json_str = '{"students" : ["Alice", "Bob"]}', random_seed = 666, max_output_length = 128, probability = 0.9)
LIMIT 3

Query id: 7f802052-efb0-41b4-87fa-03b7dd290e9d

┌─json──────────────────────────────────────────────────────────────────────────────────┐
│ {"ade0yX":[9200406572736542991, true, "sm"]}                                          │
│ {"students":["Alice", "eSN3WO#a6NYTBe0$#OWwyIQ"], "cVoP2BuQugQ":17384271928263249403} │
│ {"students":["Alice", "Bob", "T+-k4+PJGkL%XRRaF2BoeN@&A"]}                            │
└───────────────────────────────────────────────────────────────────────────────────────┘
```

Next step:
* Generate invalid string

Fixes #35962

add Object('json')

use named collection
 2023-11-28 01:58:24 +00:00
Han Fei
7bd2ea3add Merge branch 'master' into hanfei/statistic 2023-11-27 23:31:23 +01:00
Antonio Andelic
544f6081eb
Merge pull request #57259 from ClickHouse/ignore-column-comments 
Ignore comments when comparing column descriptions
 2023-11-27 20:43:39 +01:00
Nikolai Kochetov
e1b00b3ee0 Merge branch 'master' into SystemTablesFilterEngine 2023-11-27 18:54:21 +00:00
Antonio Andelic
9077550bb1 Better Readonly metric 2023-11-27 14:25:45 +00:00
Antonio Andelic
9707796869 Ignore comments when comparing column descriptions 2023-11-27 12:25:41 +00:00
vdimir
bbc202a891
Merge pull request #56816 from AVMusorin/improve-settings-rocksdb 
Show total_bytes and total_rows in system tables for RocksDB storage
 2023-11-27 11:42:15 +01:00
Kruglov Pavel
b10e46b2bc
Merge pull request #57126 from Avogar/size-virtual-column 
Add _size virtual column to s3/file/hdfs/url/azureBlobStorage engines
 2023-11-27 10:12:18 +01:00
flynn
aa15a4e46d Merge branch 'master' of github.com:ClickHouse/ClickHouse into alias_column_in_indices 2023-11-27 03:52:00 +00:00
flynn
f995ac87c5 review fix 2023-11-27 03:50:34 +00:00
Azat Khuzhin
4a02de4674 Add ability to disable checksums for S3 to avoid excessive input file read 
AWS S3 client can read file multiple times, this is required for:
- calculate checksums
- calculate signature (done only for HTTP, since ClickHouse uses
  PayloadSigningPolicy::Never)

So this means that for HTTP, to send file to S3 it will be read 3x
times, and for HTTPS 2x times.

By overriding GetChecksumAlgorithmName() to return empty string,
checksums can be disabled, and the input file will be read only once.

And even though additional https layer adds extra integrity layer,
someone still may find this too risky I guess, even though ClickHouse
internal format (for MergeTree) has checksums, and more.

Here is an example stacktrace of this excessive read:

<details>

<summary>stacktrace</summary>

    (lldb) bt
    * thread 383, name = 'BackupWorker', stop reason = breakpoint 1.1
      * frame 0: 0x00000000103c5fc0 clickhouse`DB::StdStreamBufFromReadBuffer::seekpos() + 32 at StdStreamBufFromReadBuffer.cpp:67
        frame 1: 0x000000001777f7f8 clickhouse`std::__1::basic_istream<char, std::__1::char_traits<char>>::tellg() [inlined] std::__1::basic_streambuf<char, std::__1::char_traits<char>>::pubseekoff[abi:v15000](this=<unavailable>, __off=0, __way=cur, __which=8) + 120 at streambuf:162
        frame 2: 0x000000001777f7e3 clickhouse`std::__1::basic_istream<char, std::__1::char_traits<char>>::tellg() + 99 at istream:1249
        frame 3: 0x00000000152e4979 clickhouse`Aws::Utils::Crypto::MD5OpenSSLImpl::Calculate() + 57 at CryptoImpl.cpp:223
        frame 4: 0x00000000152dedee clickhouse`Aws::Utils::Crypto::MD5::Calculate() + 14 at MD5.cpp:30
        frame 5: 0x00000000152db5ac clickhouse`Aws::Utils::HashingUtils::CalculateMD5() + 44 at HashingUtils.cpp:235
        frame 6: 0x000000001528b97b clickhouse`Aws::Client::AWSClient::AddChecksumToRequest() const + 507 at AWSClient.cpp:772
        frame 7: 0x000000001528ded2 clickhouse`Aws::Client::AWSClient::BuildHttpRequest() const + 1682 at AWSClient.cpp:930
        frame 8: 0x00000000100b864f clickhouse`DB::S3::Client::BuildHttpRequest() const + 15 at Client.cpp:622
        frame 9: 0x0000000015286a41 clickhouse`Aws::Client::AWSClient::AttemptOneRequest(this=0x00007ffde2f8f000, httpRequest=<unavailable>, request=<unavailable>, signerName=<unavailable>, signerRegionOverride=<unavailable>, signerServiceNameOverride="s3") const + 65 at AWSClient.cpp:491
        frame 10: 0x00000000152845b9 clickhouse`Aws::Client::AWSClient::AttemptExhaustively(this=0x00007ffde2f8f000, uri=0x00007ffdd4d44f38, request=0x00007ffdd4d45d10, method=HTTP_PUT, signerName="SignatureV4", signerRegionOverride="us-east-1", signerServiceNameOverride="s3") const + 1337 at AWSClient.cpp:272
        frame 11: 0x0000000015298d0d clickhouse`Aws::Client::AWSXMLClient::MakeRequest() const + 45 at AWSXmlClient.cpp:99
        frame 12: 0x0000000015298cb5 clickhouse`Aws::Client::AWSXMLClient::MakeRequest() const + 309 at AWSXmlClient.cpp:66
        frame 13: 0x0000000015354b23 clickhouse`Aws::S3::S3Client::PutObject(this=0x00007ffde2f8f000, request=0x00007ffdd4d45d10) const + 2659 at S3Client.cpp:1731
        frame 14: 0x00000000100b174f clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const [inlined]
        frame 15: 0x00000000100b173a clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const + 41 at Client.cpp:578
        frame 16: 0x00000000100b1711 clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const + 981 at Client.cpp:508
        frame 17: 0x00000000100b133c clickhouse`DB::S3::Client::PutObject(DB::S3::ExtendedRequest<Aws::S3::Model::PutObjectRequest> const&) const [inlined]
        frame 18: 0x00000000100b133c clickhouse`DB::S3::Client::PutObject() const + 28 at Client.cpp:418
        frame 19: 0x00000000103b96d6 clickhouse`DB::copyDataToS3File()

</details>

This new behaviour could be enabled with `s3_disable_checksum=true`.

Note, that I've checked this implementation with GCS/R2/S3/MinIO and it
works everywhere.
 2023-11-26 19:20:19 +01:00
Alexey Milovidov
a9bb76378f
Merge pull request #56724 from canhld94/ch_replicated_column_mismatch 
Do not allow tables on different replicas have different aggregate functions in SimpleAggregateFunction column
 2023-11-26 17:20:10 +01:00
Alexey Milovidov
1cc33f3430
Merge pull request #56164 from azat/not-byte-identical-message 
Add more details to "Data after merge is not byte-identical to data on another replicas"
 2023-11-26 04:14:15 +01:00
Alexey Milovidov
304d6375be
Merge pull request #56225 from azat/rocksdb-compact 
Allow manual compaction of rocksdb via OPTIMIZE query
 2023-11-26 03:59:54 +01:00
Robert Schulze
5b7d2a903d
Merge pull request #57152 from rschu1ze/fine-granular-plan-opt-settings 
Fine-granular enablement/disabling of plan-level optimizations
 2023-11-25 20:44:23 +01:00
flynn
9f174b1c5e Support use alias column in indices 2023-11-25 15:40:22 +00:00
Vitaly Baranov
2e7f314599
Merge pull request #50209 from ilejn/merge_row_policy 
Engine Merge obeys row policy
 2023-11-25 10:34:22 +01:00
Sema Checherinda
bd36dce81b
Merge pull request #57129 from ClickHouse/fix_something 
Fix something in ReplicatedMergeTree
 2023-11-24 20:19:25 +01:00
Alexander Tokmakov
deda8483be
Merge pull request #56536 from ClickHouse/move-partition-inactive-replica 
Fix race between REPLACE_RANGE and GET_PART (set actual part name when fetching)
 2023-11-24 20:09:24 +01:00
Aleksandr Musorin
c3b617bd7b Move class functions upper 2023-11-24 19:00:00 +01:00
Raúl Marín
63fe821f7b Merge remote-tracking branch 'blessed/master' into backup_1 2023-11-24 16:21:59 +01:00
Raúl Marín
2539100afb Review improvements 2023-11-24 16:21:56 +01:00
Kseniia Sumarokova
13187bf4f1
Merge pull request #56959 from melvynator/master 
Add experimental flag for MaterializedPostgreSQL table engine.
 2023-11-24 16:16:42 +01:00
Kseniia Sumarokova
c25c6ec403
Merge pull request #57031 from aiven-sal/aiven-sal/fix_namedcoll2 
NamedCollections: make exception message more informative.
 2023-11-24 16:15:16 +01:00
alesapin
a3c2896e9d
Merge pull request #56846 from aalexfvk/add_cancellation_point_for_moving_background_operation 
Add cancellation hook for moving background operation
 2023-11-24 12:12:47 +01:00
Chen Lixiang
2b35e98ef8 add comments and doc 2023-11-24 16:40:19 +08:00
Chen Lixiang
89fa729235 show uncompressed size in system.tables 2023-11-24 15:54:27 +08:00
Alexey Milovidov
81a50703c3 Fix tests 2023-11-23 23:39:40 +01:00
Alexey Milovidov
ade67cc57e Merge branch 'master' of github.com:ClickHouse/ClickHouse into CurtizJ-fix-ttl-non-deterministic 2023-11-23 23:04:11 +01:00
Alexey Milovidov
d72bc854d1
Merge pull request #57074 from ClickHouse/less-allocation-in-arenas 
Avoid excessive allocation in Arena
 2023-11-23 23:02:26 +01:00
Igor Nikonov
0434044a7e
Merge pull request #57054 from ClickHouse/pr-cleanup-narrow-dependency 
Parallel replicas: cleanup, narrow dependency
 2023-11-23 20:12:37 +01:00
Robert Schulze
77d48e4d55
"optimise" --> "optimize". Sorry, friends from the UK. 2023-11-23 18:41:35 +00:00
Alexey Milovidov
c5fabcc472 Fix tests 2023-11-23 17:50:50 +01:00
Alexey Milovidov
dc0d6e432c Fix build 2023-11-23 16:11:54 +01:00
Alexey Milovidov
6bbf449808 Fix build 2023-11-23 16:06:35 +01:00
Alexey Milovidov
39ea0c426a Fix build 2023-11-23 16:05:15 +01:00
Alexey Milovidov
89656723fb Fix tests 2023-11-23 16:02:33 +01:00
Alexey Milovidov
93b789169b Merge branch 'master' into CurtizJ-fix-ttl-non-deterministic 2023-11-23 15:46:01 +01:00
Aleksei Filatov
1a03f5f7f4 Merge remote-tracking branch 'upstream/master' into add_cancellation_point_for_moving_background_operation 2023-11-23 16:43:33 +03:00
serxa
663c8cd944 Merge branch 'master' into dashboards-table 2023-11-23 10:29:48 +00:00
serxa
0fc25fbda8 Merge branch 'master' into async-loader-integration 2023-11-23 09:32:43 +00:00
Alexander Tokmakov
a0ce9f46cf fix 2023-11-22 23:15:20 +01:00
Alexey Milovidov
2ba2f48051 Merge branch 'master' of github.com:ClickHouse/ClickHouse into less-allocation-in-arenas 2023-11-22 22:11:59 +01:00
avogar
70f45c11bd Fix 2023-11-22 19:56:42 +00:00
Alexander Tokmakov
ef3e558a65 fix no such key for empty parts 2023-11-22 20:51:10 +01:00
Alexander Tokmakov
d9eb68ba35 fix unexpected outdated parts 2023-11-22 20:29:57 +01:00
Alexander Tokmakov
9f1a9826af fix replace/move partition 2023-11-22 20:04:14 +01:00
Alexander Tokmakov
86f7732c24
Update StorageReplicatedMergeTree.cpp 2023-11-22 19:41:27 +01:00
avogar
031f030102 Remove unused method 2023-11-22 18:27:32 +00:00
Kruglov Pavel
b84e3cf683
Merge branch 'master' into size-virtual-column 2023-11-22 19:25:00 +01:00
avogar
4a86f4a7b9 Fix style changes 2023-11-22 18:24:34 +00:00
avogar
007353a2dd Add _size virtual column to s3/file/hdfs/url/azureBlobStorage engines 2023-11-22 18:12:36 +00:00
Alexander Tokmakov
d277bfb225 setActualPartName for executeReplaceRange 2023-11-22 18:48:25 +01:00
vdimir
ffbe85d3a0
Merge pull request #56668 from ClickHouse/vdimir/analyzer_s3_partition_pruning 
Analyzer: filtering by virtual columns for StorageS3
 2023-11-22 16:44:44 +01:00
Duc Canh Le
44a3dc1cd8 add new method to check if two types are exactly identical 
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2023-11-22 15:02:34 +00:00
Dmitry Novik
7539928814
Merge pull request #56079 from ZhiguoZh/20231027-combine-filter-avx512 
Optimize DB::combineFilters with AVX512_VBMI2 intrinsic
 2023-11-22 15:42:33 +01:00
zvonand
5153798aeb Introduced fileCluster table function 
Added fileCluster function
Added test and docs
 2023-11-22 15:06:04 +01:00
Kruglov Pavel
9fb1acc947
Merge pull request #56810 from Avogar/iceberg-metadata-files 
Iceberg metadata files
 2023-11-22 14:40:04 +01:00
Alexey Milovidov
9cf7ebeced Minor changes 2023-11-22 12:13:47 +01:00
Raúl Marín
281060329f Merge remote-tracking branch 'blessed/master' into backup_1 2023-11-22 10:35:18 +01:00
Ilya Golshtein
c2816ec589 merge_row_policy: make clang-tidy happy 2023-11-21 22:06:18 +00:00
Dmitry Novik
1e0b4f979c
Add extra test 2023-11-21 22:55:55 +01:00
Kseniia Sumarokova
04aa0b10fc
Merge pull request #56990 from ClickHouse/s3queue-fix-incorrect-ref-count-from-exception-in-ctor 
S3Queue: fix metadata reference increment
 2023-11-21 21:37:23 +01:00
Alexander Tokmakov
990e3ea1c8 Merge branch 'master' into rmt_check_shutdown_flags_in_retry_loops 2023-11-21 18:01:16 +01:00
vdimir
a139ae97eb
Merge pull request #52918 from ClickHouse/vdimir/s3_blob_log 
Add system table with blob storage operations log
 2023-11-21 17:40:42 +01:00
Igor Nikonov
63a1625b77 Merge remote-tracking branch 'origin/master' into pr-cleanup-narrow-dependency 2023-11-21 16:05:48 +00:00
Igor Nikonov
ce98dfb251 Settings pointer to reference 2023-11-21 16:04:54 +00:00
Igor Nikonov
66f6a6575f Cleanup iteration: settings usage 2023-11-21 13:29:04 +00:00
Igor Nikonov
83c6ac4f6c Remove storage as parameter dependency, only columns are needed 2023-11-21 12:14:12 +00:00
alesapin
227bb30d02
Merge pull request #56986 from kevinmingtarja/implicit-constraint-collapsing-merged-tree 
Add implicit constraint for CollapsingMergeTree sign column
 2023-11-21 12:43:51 +01:00
kssenii
474f1ae0b6 Better 2023-11-21 11:42:23 +01:00
Aleksandr Musorin
ee0ae17ce2 Added optimize_trivial_approximate_count_query documentatiosn for RocksDB 2023-11-21 11:31:56 +01:00
Salvatore Mesoraca
f96cd4cfb8
Update src/Storages/NamedCollectionsHelpers.cpp 
Co-authored-by: Nikolay Degterinsky <43110995+evillique@users.noreply.github.com>
 2023-11-21 11:14:48 +01:00
Salvatore Mesoraca
970e34f32e
NamedCollections: make exception message more informative. 
Fix test 00002_log_and_exception_messages_formatting
 2023-11-21 10:21:28 +01:00
vdimir
15234474d7
Implement system table blob_storage_log 2023-11-21 09:18:25 +00:00
Amos Bird
ee1cec4d38
Allow RENAME of index columns. 2023-11-21 15:42:54 +08:00
Amos Bird
6581aaf9e8
Prevent incompatible ALTER of projection columns. 2023-11-21 15:42:54 +08:00
kevinyhzou
3adc8fdf78 Fix ci 2023-11-21 11:22:12 +08:00
Raúl Marín
04f966c471 Recover special handling of ephemeral nodes in ZooKeeperWithFaultInjection 2023-11-20 22:17:34 +01:00
Kseniia Sumarokova
2beee5d9cb
Merge branch 'master' into s3queue-fix-empty-key-check 2023-11-20 20:17:47 +01:00
jsc0218
3d5265c1d3 Merge remote-tracking branch 'origin/master' into SystemTablesFilterEngine 
merge master
 2023-11-20 18:10:54 +00:00
vdimir
31a6c7c1c4
Style changes around filterKeysForPartitionPruning 2023-11-20 18:08:45 +00:00
vdimir
95e9a27417
Remove ast based code from filterKeysForPartitionPruning 2023-11-20 17:59:58 +00:00
vdimir
a915eeded8
StorageS3 use filters from SourceStepWithFilter 2023-11-20 17:59:58 +00:00
vdimir
1cddfb1e6b
rewrite getBlockWithVirtuals for S3Storage 2023-11-20 17:59:57 +00:00
vdimir
cbb2e02c03
Analyzer: partition pruning for S3 2023-11-20 17:59:53 +00:00
Aleksandr Musorin
38133692bb Add total_bytes for RocksDB storage 
Show total_bytes in system tables for RocksDB storage

Previously it did not work because total_rows was calculated only when optimize_trivial_approximate_count_query=1
 2023-11-20 17:08:35 +01:00
Kevin Mingtarja
1d668264e1 Fix style 2023-11-20 23:54:36 +08:00
Kevin Mingtarja
d84d5692ef Enable implicit constraint for VersionedCollapsing 2023-11-20 23:52:22 +08:00
Zhiguo Zhou
be0f0b6879 Fix code style 2023-11-20 23:34:52 +08:00
kssenii
9b5e180995 Fix 2023-11-20 16:27:02 +01:00
avogar
f537bad469 Merge branch 'master' of github.com:ClickHouse/ClickHouse into schema-inference-union 2023-11-20 14:32:50 +00:00
avogar
872556a5d4 Merge branch 'master' of github.com:ClickHouse/ClickHouse into schema-inference-union 2023-11-20 14:03:36 +00:00
Sema Checherinda
f999337dae
Revert "Revert "s3 adaptive timeouts"" 2023-11-20 14:53:22 +01:00
Alexander Tokmakov
5031f239c3
Revert "s3 adaptive timeouts" 2023-11-20 14:28:59 +01:00
Raúl Marín
8217915064 Replace part_committed_locally_but_zookeeper with retries 2023-11-20 13:55:50 +01:00
kssenii
1ade4b797b Add a comment 2023-11-20 12:48:49 +01:00
kssenii
10fb40ece9 Fix 2023-11-20 12:44:17 +01:00
alesapin
857ce2d880
Merge branch 'master' into implicit-constraint-collapsing-merged-tree 2023-11-20 12:28:55 +01:00
Raúl Marín
aadb786b2b Merge remote-tracking branch 'blessed/master' into backup_1 2023-11-20 10:53:31 +01:00
Zhiguo Zhou
1f044b11e3 Enhance unittest MergeTree.CombineFilters 2023-11-20 16:00:14 +08:00
Kevin Mingtarja
af0c954c9e Add implicit constraint for CollapsingMergeTree sign column 2023-11-20 15:05:12 +08:00
melvynator
cfdb663892 Remove a tab 2023-11-20 00:46:14 +01:00
Nikolay Degterinsky
9683adfb03
Merge pull request #56830 from evillique/better-add-column 
Fix flattening existing Nested columns during ADD COLUMN
 2023-11-19 16:21:56 +01:00
Sema Checherinda
a950595c24
Merge pull request #56314 from CheSema/s3-aggressive-timeouts 
s3 adaptive timeouts
 2023-11-19 14:12:14 +01:00
serxa
9023f61349 Merge branch 'master' into async-loader-integration 2023-11-19 13:05:58 +00:00
Alexey Milovidov
edc3b2fe48
Merge pull request #56958 from ClickHouse/metric-queued-jobs 
Add metrics for the number of queued jobs, which is useful for the IO thread pool
 2023-11-19 10:37:18 +01:00
jsc0218
0171d61fa5 Add MySQL Engine Filter 2023-11-19 02:34:14 +00:00
Sema Checherinda
c465933b1d
Merge pull request #56808 from CheSema/retry-fetch-part 
Fetching a part waits when that part is fully committed on remote replica
 2023-11-19 01:57:25 +01:00
melvynator
3ade60a261 Keep trying to fix style 2023-11-19 01:31:21 +01:00
melvynator
88b0525306 Style check 2023-11-19 01:10:25 +01:00
melvynator
5f0f9b0ad4 Test check style 2023-11-19 01:05:11 +01:00
Ubuntu
ddd9a23116 Fix newline 2023-11-19 00:02:33 +00:00
Ubuntu
89b93adf82 Fix trailing whitespace 2023-11-18 23:55:46 +00:00
Ilya Golshtein
a8abec6cac merge_row_policy: cleanup after merge 2023-11-18 21:45:17 +00:00
Antonio Andelic
274f2c71f2
Merge pull request #56460 from ClickHouse/keeper-map-backup-restore 
Backup/Restore for KeeperMap tables
 2023-11-18 19:27:36 +01:00
Ubuntu
16e7bd201b Add experimental flag for PostgresSQL table engine. Also add documentation on how to enable it 2023-11-18 18:22:33 +00:00
Alexey Milovidov
d56cbda185 Add metrics for the number of queued jobs, which is useful for the IO thread pool 2023-11-18 19:07:59 +01:00
Nikolay Degterinsky
a18b715591 Fix a bug 2023-11-18 15:44:45 +00:00
Raúl Marín
210a0ee045 Merge remote-tracking branch 'blessed/master' into backup_1 2023-11-17 18:50:24 +01:00
Sema Checherinda
29e64347b9 improve exception message 2023-11-17 17:37:14 +01:00
jsc0218
5b20118c66 Fix Typo 2023-11-17 14:55:36 +00:00
jsc0218
fb96f65904 Fix Style 2023-11-17 14:15:33 +00:00
Raúl Marín
3633e77a37 Refactor ZooKeeperWithFaultInjection 2023-11-17 12:57:32 +01:00
Sema Checherinda
86119dbc3f fix data race 2023-11-17 12:26:50 +01:00
Kseniia Sumarokova
3322f54713
Merge branch 'master' into remove-unused-code 2023-11-17 11:55:02 +01:00
Antonio Andelic
9bcedf3764 Cleanup 2023-11-17 10:27:19 +00:00
Antonio Andelic
803d9668e2 Merge branch 'master' into keeper-map-backup-restore 2023-11-17 10:11:59 +00:00
Ilya Golshtein
9eeae568e4 Merge remote-tracking branch 'origin/master' into merge_row_policy 
merge_row_policy: changes because of refactoring in master
 2023-11-17 09:19:56 +00:00
Aleksei Filatov
163b8a814e Fix clang-tidy issues and disable parallel execution of tests 2023-11-17 11:02:07 +03:00
jsc0218
bd5a715244 Fix Style 2023-11-17 03:26:36 +00:00