thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-14 10:22:10 +00:00
Code Issues Packages Projects Releases Wiki Activity
134,844 Commits 9,328 Branches 1,783 Tags 2.1 GiB
46c2f85314
Commit Graph

16209 Commits

Author SHA1 Message Date
Kseniia Sumarokova
e029021696
Merge pull request #59446 from ClickHouse/s3-queue-allow-to-add-start-point 
Allow to define a starting point for s3queue ordered mode at creation
 2024-02-01 16:46:24 +01:00
Alexey Milovidov
f52b08070f Merge branch 'master' into remove-unused-headers 2024-02-01 13:42:37 +01:00
kssenii
3a16427e00 Fix test 2024-02-01 12:35:01 +01:00
Maksim Kita
ecbea6f625 Updated implementation 2024-02-01 12:51:50 +03:00
Kseniia Sumarokova
b3418c506e
Update src/Storages/S3Queue/S3QueueSettings.h 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2024-02-01 10:28:33 +01:00
Dmitry Novik
5b630ed745
Merge pull request #59399 from azat/build/format-fwd-decl 
Forward declaration for PeekableReadBuffer
 2024-02-01 09:15:07 +01:00
Alexey Milovidov
33f6940214 CLion says these headers are unused 2024-02-01 03:50:09 +01:00
kssenii
361b2f107b Allow to define a starting point for s3queue ordered mode 2024-01-31 21:28:18 +01:00
Alexander Gololobov
5947e5b5a8 Use retry settings and fault injection 2024-01-31 16:48:46 +01:00
Dmitry Novik
572d47acc7 Remove logging 2024-01-31 14:43:06 +00:00
Dmitry Novik
0d21004218 WIP on StorageMerge 2024-01-31 13:50:15 +00:00
Kseniia Sumarokova
b20567a055
Merge pull request #59167 from ClickHouse/s3-queue-parallelize-ordered-mode 
S3Queue: allow parallel & disrtibuted processing for ordered mode
 2024-01-31 11:20:41 +01:00
Kseniia Sumarokova
caaafbcedb
Merge pull request #56864 from ClickHouse/broken-projections-better-handling 
Do not consider data part as broken if projection is broken
 2024-01-31 11:17:15 +01:00
Dmitry Novik
ac17ad1e66
Merge pull request #59073 from ClickHouse/fix-analyzer-resolve-distr-id 
Analyzer: Do not resolve remote table id on initiator
 2024-01-31 09:08:30 +01:00
Nikita Taranov
1be8b61c5b add test 2024-01-30 21:52:32 +01:00
Dmitry Novik
722f3db738 Merge remote-tracking branch 'origin/master' into storage-merge-aliases-analyzer 2024-01-30 19:50:01 +00:00
Nikita Taranov
17ab2674f4 impl 2024-01-30 20:35:10 +01:00
Azat Khuzhin
010f1c6cb7 Forward declaration for PeekableReadBuffer 
ReadHelpers.h is very common header and is the root cause of "recompile
everything".

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-30 19:24:19 +01:00
Alexander Gololobov
c348c4e828 Move ZooKeeperRetries.h to Common 2024-01-30 19:02:17 +01:00
Alexey Milovidov
4ddba907c8 Update autogenerated version to 24.2.1.1 and contributors 2024-01-30 18:49:28 +01:00
Alexander Gololobov
82c06ca294 Use ZooKeeperRetriesControl 2024-01-30 18:31:37 +01:00
Alexander Gololobov
023b8cbd53 Retry disconnects and expired sessions 2024-01-30 17:47:11 +01:00
kssenii
145f6a31cb Fix 2024-01-30 16:29:40 +01:00
Kruglov Pavel
3105960fd8
Merge pull request #59352 from Avogar/fix-bad-attached-rabbitmq 
Fix lazy initialization in RabbitMQ
 2024-01-30 12:52:07 +01:00
Kseniia Sumarokova
e16aaa8d81
Merge branch 'master' into broken-projections-better-handling 2024-01-30 10:54:22 +01:00
Dmitry Novik
5a34641e91 Merge remote-tracking branch 'origin/master' into fix-analyzer-resolve-distr-id 2024-01-30 07:17:15 +00:00
Alexey Milovidov
35ea956255
Merge pull request #59295 from ClickHouse/fix_block_number 
Fix `_block_number` column with non-default serializations and mutation
 2024-01-30 04:07:09 +01:00
Alexey Milovidov
ac90637170
Merge pull request #59122 from arthurpassos/bring_back_attach_partition_pr 
Revert revert attach partition PR
 2024-01-30 04:02:31 +01:00
Alexey Milovidov
de341a6d6e
Merge pull request #59351 from ClickHouse/fix-non-redy-set-system-tables 
Fix not-ready set for system.tables
 2024-01-30 03:59:18 +01:00
Alexey Milovidov
50532c485f
Merge pull request #59340 from CurtizJ/lower-memory-usage-vertical-merges 
Lower memory usage in vertical merges
 2024-01-30 03:42:28 +01:00
avogar
3a5ba56c36 Fix lazy initialization in RabbitMQ, fix possible deadlock on insert into unitialized queue engine 2024-01-29 20:09:09 +00:00
Robert Schulze
2cfcaf8ff4
Merge pull request #59040 from MochiXu/fix_remove_inverted_index_files 
`DROP INDEX` of inverted index now removes all relevant files from persistence
 2024-01-29 20:39:56 +01:00
kssenii
df7d3b8a40 Allocate current shard number from keeper 2024-01-29 19:36:16 +01:00
Arthur Passos
15d04d0a78
Merge branch 'master' into bring_back_attach_partition_pr 2024-01-29 15:16:49 -03:00
Arthur Passos
516daadeef add static storage assert in datapartcloner 2024-01-29 15:16:00 -03:00
Nikolai Kochetov
d0143380d7 Fix not-ready set for system.tables 2024-01-29 17:53:59 +00:00
Nikolai Kochetov
5b2648929b
Merge pull request #59273 from kitaisreal/atomic-logger 
Added AtomicLogger
 2024-01-29 16:49:18 +01:00
avogar
377937d415 Merge branch 'master' of github.com:ClickHouse/ClickHouse into auto-format-detection 2024-01-29 15:45:18 +00:00
Anton Popov
d3cdc8826e lower memory usage in vertical merges 2024-01-29 14:06:46 +00:00
kssenii
d0ea5392db Fix storing iterated keys 2024-01-29 12:36:32 +01:00
kssenii
ef252d8376 Validate settings with keeper, add a test 2024-01-29 12:36:00 +01:00
Kruglov Pavel
6858d2f4ca
Merge pull request #58047 from Avogar/variant-data-type 
Implement Variant data type
 2024-01-29 11:36:08 +01:00
Smita Kulkarni
3183026521 Merge branch 'master' into Azure_backup 2024-01-29 11:08:13 +01:00
Kseniia Sumarokova
a29ab8f29c
Merge pull request #59170 from azat/disks/s3-plain-improvements 
Small improvements for tables on write-once (s3_plain)/read-only (web) disks
 2024-01-29 10:44:37 +01:00
Kseniia Sumarokova
0f88da5fd4
Merge pull request #59275 from ClickHouse/Avogar-patch-1 
Fix abort in iceberg metadata on bad file paths
 2024-01-29 10:35:34 +01:00
alesapin
b864b122fa Merge branch 'master' into fix_block_number 2024-01-29 10:19:43 +01:00
Alexey Milovidov
2cb2bcfbc3
Merge pull request #59315 from ClickHouse/support-backups-for-compressed-memory-tables 
Support backups for compressed in-memory tables
 2024-01-29 01:50:41 +01:00
Alexey Milovidov
aec3f28ccb Support backups for compressed in-memory tables 2024-01-28 23:06:50 +01:00
Arthur Passos
0bbfef884c
Merge branch 'ClickHouse:master' into bring_back_attach_partition_pr 2024-01-28 17:43:38 -03:00
Anton Popov
ef133bfd47 Merge remote-tracking branch 'upstream/master' into lower-memory-usage 2024-01-28 15:07:44 +00:00
avogar
92db86b829 Merge branch 'master' of github.com:ClickHouse/ClickHouse into variant-data-type 2024-01-28 14:33:49 +00:00
Maksim Kita
07750ee949 Added AtomicLogger 2024-01-28 13:55:44 +03:00
alesapin
7022ca9ece Fix _block_number column with non-default serializations and mutation 2024-01-27 20:24:03 +01:00
Anton Popov
9d3b4e5308 lower memory usage 2024-01-27 16:40:11 +00:00
Azat Khuzhin
d94547d338 Use fresh ZooKeeper client on DROP (to have higher chances on success) 
In case of DROP the client can be expired, and even though
StorageReplicatedMergeTree::drop() uses getZooKeeperIfTableShutDown(),
which creates new client if current is expired, it is not enough, since
current_zookeeper (cached client) will be used implicitly from
dropAllData().

This had been found by stress tests, that leads to DROP query hang [1].

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/59255/94eb33ef27a9ab7c4a99af40772ea287e67efcbf/stress_test__tsan_.html

<details>

<summary>stacktrace</summary>

    2024.01.26 17:41:41.138577 [ 3319 ] {} <Error> DatabaseCatalog: Cannot drop table test_vzfk1xoc.alter_table1 (455a674c-161d-44d8-abc2-fd205bad1116). Will retry later.: Code: 999. Coordination::Exception: Session expired. (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

    0. ./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000001e06a5e3 in /usr/bin/clickhouse
    1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f697c74 in /usr/bin/clickhouse
    2. ./src/Common/Exception.h:0: Coordination::Exception::Exception<char const (&) [16]>(char const (&) [16], Coordination::Error) @ 0x000000001adc298b in /usr/bin/clickhouse
    3. ./src/Common/ZooKeeper/IKeeper.h:0: Coordination::ZooKeeper::pushRequest(Coordination::ZooKeeper::RequestInfo&&) @ 0x000000001adfb418 in /usr/bin/clickhouse
    4. ./build_docker/./src/Common/ZooKeeper/ZooKeeperImpl.cpp:1343: Coordination::ZooKeeper::get(String const&, std::function<void (Coordination::GetResponse const&)>, std::shared_ptr<std::function<void (Coordination::WatchResponse const&)>>) @ 0x000000001adfd8e5 in /usr/bin/clickhouse
    5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x000000001ad890bd in /usr/bin/clickhouse
    6. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x000000001ad88a36 in /usr/bin/clickhouse
    7. ./build_docker/./src/Common/ZooKeeper/ZooKeeper.cpp:580: zkutil::ZooKeeper::tryGetWatch(String const&, String&, Coordination::Stat*, std::function<void (Coordination::WatchResponse const&)>, Coordination::Error*) @ 0x000000001ad898bf in /usr/bin/clickhouse
    8. ./build_docker/./src/Common/ZooKeeper/ZooKeeper.cpp:570: zkutil::ZooKeeper::tryGet(String const&, String&, Coordination::Stat*, std::shared_ptr<Poco::Event> const&, Coordination::Error*) @ 0x000000001ad89554 in /usr/bin/clickhouse
    9. ./build_docker/./src/Common/ZooKeeper/ZooKeeperWithFaultInjection.cpp:0: DB::ZooKeeperWithFaultInjection::tryGet(String const&, String&, Coordination::Stat*, std::shared_ptr<Poco::Event> const&, Coordination::Error*) @ 0x000000001ae110dc in /usr/bin/clickhouse
    10. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:701: DB::StorageReplicatedMergeTree::unlockSharedDataByID(String, String const&, DB::MergeTreePartInfo const&, String const&, String const&, std::shared_ptr<DB::ZooKeeperWithFaultInjection> const&, DB::MergeTreeSettings const&, std::shared_ptr<Poco::Logger>, String const&, StrongTypedef<unsigned int, DB::MergeTreeDataFormatVersionTag>) @ 0x000000001975527e in /usr/bin/clickhouse
    11. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::removeSharedDetachedPart(std::shared_ptr<DB::IDisk>, String const&, String const&, String const&, String const&, String const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<zkutil::ZooKeeper> const&) @ 0x000000001976c980 in /usr/bin/clickhouse
    12. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:10171: DB::StorageReplicatedMergeTree::removeDetachedPart(std::shared_ptr<DB::IDisk>, String const&, String const&) @ 0x000000001976bdc6 in /usr/bin/clickhouse
    13. ./build_docker/./src/Storages/MergeTree/MergeTreeData.cpp:2798: DB::MergeTreeData::dropAllData() @ 0x0000000019c643c8 in /usr/bin/clickhouse
    14. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:1117: DB::StorageReplicatedMergeTree::drop() @ 0x0000000019664e0a in /usr/bin/clickhouse
    15. ./build_docker/./src/Interpreters/DatabaseCatalog.cpp:0: DB::DatabaseCatalog::dropTableFinally(DB::DatabaseCatalog::TableMarkedAsDropped const&) @ 0x0000000017bf1fac in /usr/bin/clickhouse
    16. ./build_docker/./src/Interpreters/DatabaseCatalog.cpp:0: DB::DatabaseCatalog::dropTableDataTask() @ 0x0000000017bf175e in /usr/bin/clickhouse
    17. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x0000000017bf94c2 in /usr/bin/clickhouse
    18. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000170bd0b8 in /usr/bin/clickhouse
    19. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:701: DB::BackgroundSchedulePool::threadFunction() @ 0x00000000170c06ce in /usr/bin/clickhouse
    20. ./build_docker/./src/Core/BackgroundSchedulePool.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000170c0feb in /usr/bin/clickhouse
    21. ./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f7c6a2f in /usr/bin/clickhouse
    22. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f7cc8f2 in /usr/bin/clickhouse
    23. ? @ 0x000000000733904f in /usr/bin/clickhouse
    24. ? @ 0x00007f28b2319ac3
    25. ? @ 0x00007f28b23ab850

</details>

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-27 16:04:18 +01:00
kssenii
0a21e6aa5d Fix 2024-01-26 19:07:30 +01:00
avogar
5833641fa5 Merge branch 'master' of github.com:ClickHouse/ClickHouse into variant-data-type 2024-01-26 16:54:02 +00:00
Kruglov Pavel
8fcedddcda
Fix abort in iceberg metadata on bad file paths 2024-01-26 17:46:49 +01:00
Azat Khuzhin
90ab986be6 Revert "Prohibit CREATE on static storages (ATTACH should be used instead)" 
CREATE is used by Replicated database even for ATTACH, and anyway ATTACH
creates format_version.txt as well (without previuos patch).

This reverts commit 323f1f320ebe7e588d443abf78efa2c60193e7a9.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-26 16:47:36 +01:00
Azat Khuzhin
7ba31bf55c Do not create format_version.txt on write-once storages 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-26 16:47:36 +01:00
Azat Khuzhin
66a40bda0a Revert "Prohibit DROP from static storages" 
It is actually OK to DROP at least from read-only disk, as for the
write-once it should be OK as well, since CREATE is prohibited anyway,
so there should be no leftovers

CI: https://s3.amazonaws.com/clickhouse-test-reports/59170/c82050d1925439f0ede2b32acb5b1b8df4acae5d/stress_test__asan_.html
CI: https://s3.amazonaws.com/clickhouse-test-reports/59170/c82050d1925439f0ede2b32acb5b1b8df4acae5d/integration_tests__asan__[3_4].html

This reverts commit 082e62712c5e2f4c48d1abc6141ccfd1249d26b4.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-26 16:47:35 +01:00
Azat Khuzhin
aafc149c1e Prohibit CREATE on static storages (ATTACH should be used instead) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-26 16:47:35 +01:00
Azat Khuzhin
1ee723d50d Prohibit DROP from static storages 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-26 16:47:35 +01:00
Azat Khuzhin
f3b4f28f0d Disable mutable operations into {Replicated,}MergeTree with static storage 
Mutable operations includes:
- INSERT
- OPTIMIZE
- parts manipulation
- ...

But does not include DROP, since you need an ability to DROP it.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-26 16:47:35 +01:00
Azat Khuzhin
6751e01cc3 Do not drop parts for static (ro/write-once) storages for Replicated engine 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-26 16:47:35 +01:00
kssenii
5cd757dd9a Merge remote-tracking branch 'origin/master' into s3-queue-parallelize-ordered-mode 2024-01-26 16:33:22 +01:00
Kseniia Sumarokova
99653fc778
Merge branch 'master' into broken-projections-better-handling 2024-01-26 16:27:23 +01:00
kssenii
ee15b6e646 Fix 2024-01-26 16:26:59 +01:00
Kruglov Pavel
946c4e0495
Merge branch 'master' into auto-format-detection 2024-01-26 15:51:35 +01:00
Kruglov Pavel
692d37306e
Fix build 2024-01-26 15:50:16 +01:00
Arthur Passos
ce6877cbb6
Merge branch 'ClickHouse:master' into bring_back_attach_partition_pr 2024-01-26 10:57:11 -03:00
Kruglov Pavel
46a6b84a5a
Merge branch 'master' into auto-format-detection 2024-01-25 22:11:07 +01:00
Maksim Kita
5bb734a4bb ActionsDAG buildFilterActionsDAG refactoring 2024-01-25 18:24:14 +03:00
Maksim Kita
2a327107b6 Updated implementation 2024-01-25 14:31:49 +03:00
Maksim Kita
42029f42e7 Table CREATE DROP Poco::Logger memory leak fix 2024-01-25 14:31:18 +03:00
Kseniia Sumarokova
951c874aa9
Merge pull request #59080 from ClickHouse/data-lakes-survive-restart 
Allow server to start with broken data lake table and fix itself lazily
 2024-01-25 11:50:16 +01:00
Nikita Taranov
1e3f6894a3
Merge pull request #59089 from nickitat/sort_res_ranges_before_sending_to_replica 
Coordinator returns ranges for reading in sorted order
 2024-01-24 21:04:53 +01:00
avogar
11f1ea50d7 Fix tests 2024-01-24 17:55:31 +00:00
kssenii
5ecf36f275 Fix style check 2024-01-24 17:19:52 +01:00
Dmitry Novik
95e2a3d6da
Merge pull request #58911 from ClickHouse/fix-index-hint-crash 
Fix crash when indexHint() is used
 2024-01-24 17:16:22 +01:00
kssenii
7f8f379d7f Parallel & disrtibuted processing for ordered mode 2024-01-24 16:32:15 +01:00
Kruglov Pavel
89bafbcda6
Merge pull request #59133 from Avogar/allow-to-ignore-schema-evolution-in-iceberg 
Allow to ignore schema evolution in Iceberg table engine under a setting
 2024-01-24 13:19:09 +01:00
Smita Kulkarni
4d92eba316 Merge branch 'master' into Azure_backup 2024-01-24 11:13:57 +01:00
avogar
f05174e441 Fix style 2024-01-23 18:59:40 +00:00
avogar
617cc514b7 Try to detect file format automatically during schema inference if it's unknown 2024-01-23 18:59:39 +00:00
Arthur Passos
dba792dff0
Remove duplicate includ 2024-01-23 15:03:31 -03:00
Arthur Passos
e6b17377d0
Merge branch 'master' into bring_back_attach_partition_pr 2024-01-23 14:52:47 -03:00
avogar
be3b5dc45f Allow to ignore schema evolution in Iceberg table engine under a setting 2024-01-23 17:45:41 +00:00
Nikolai Kochetov
d64b491b45 Fixing build 2024-01-23 17:00:21 +00:00
Nikolai Kochetov
a26f8f5425
Merge pull request #59076 from ClickHouse/revert-59059-revert-58838-allow-parallel-replicas-for-join-with-analyzer 
Revert "Revert "Allow parallel replicas for JOIN with analyzer [part 1].""
 2024-01-23 17:01:19 +01:00
Arthur Passos
5179891aef remove static log 2024-01-23 11:43:08 -03:00
Arthur Passos
799a94081b Revert "Revert "Allow to attach partition from table with different partition expression when destination partition expression doesn't re-partition"" 
This reverts commit a1c83e2f51.
 2024-01-23 11:40:42 -03:00
Robert Schulze
a1c83e2f51
Revert "Allow to attach partition from table with different partition expression when destination partition expression doesn't re-partition" 2024-01-23 15:26:04 +01:00
Anton Popov
4292ee7d51
Merge pull request #58886 from CurtizJ/better-read-buffers 
Less error prone interface of read buffers
 2024-01-23 14:59:27 +01:00
Alexander Gololobov
9a0e5d0aac
Merge pull request #58914 from ClickHouse/ot_spans_in_insert 
Opentelemetry spans to analyze CPU and S3 bottlenecks on inserts
 2024-01-23 13:53:13 +01:00
Nikolai Kochetov
fcbab407b8
Merge branch 'master' into revert-59059-revert-58838-allow-parallel-replicas-for-join-with-analyzer 2024-01-23 12:21:44 +01:00
kssenii
bca73d4b0a Fix 2024-01-23 11:51:21 +01:00
kssenii
b76630c955 Merge remote-tracking branch 'origin/master' into broken-projections-better-handling 2024-01-23 11:34:58 +01:00
Robert Schulze
e67076ea2b
Merge pull request #55201 from canhld94/column_level_compress_block 
Column-level compression block sizes
 2024-01-23 11:22:03 +01:00
Robert Schulze
bfec324b28
Some fixups + test 2024-01-23 10:17:16 +00:00
Robert Schulze
81543c0a46
Merge remote-tracking branch 'rschu1ze/master' into fix_remove_inverted_index_files 2024-01-23 09:45:09 +00:00
Nikita Taranov
09e24ed6c5
Merge pull request #58068 from zhongyuankai/limit_materialized_view_size 
A limit on the number of materialized views attached to a table.
 2024-01-22 22:50:56 +01:00
Nikita Taranov
b161921825 impl 2024-01-22 22:33:23 +01:00
Max K
4932b1cad9
Merge branch 'master' into better-read-buffers 2024-01-22 21:52:55 +01:00
Arthur Passos
24b8bbe9fa
Allow to attach partition from table with different partition expression when destination partition expression doesn't re-partition (#39507) 
* temp commit

* temp commit

* draft impl for feedback

* fix weird style changes

* fix weird style changes

* fix weird style changes

* fix weird style changes

* fix weird style changes

* aa

* aa

* Add integ tests and remove partition key restriction

* fix small incosistency in partition id

* style fix

* style fix

* style fix

* use existing DataPartStorageBuilder instead of new one

* Refactor part clone to make it more readable and maintainable

* Add MergeTreeDataPartCloner docs

* define ErrorCodes::BAD_ARGUMENTS

* Rebase

* camel case methods

* address some comments

* yet another rebase?

* Move from integ tests to stateless tests

* address more comments

* add finalize on min_max_idx files

* Add sync option to DistinctPartitionExpCloner

* just a temp test

* revert temp change

* Use echoOn to distinguish test queries

* remove comment

* fix build issue during rebase

* atempt to fix build after rebase

* finally fix build

* clear minmaxidx hyperrectangle before loading it

* Fix error on min_max files deletion where it was being assumed that partition expression contained all columns

* get it to the state it was previously

* add missing include

* getting functional?

* refactoring and renaming

* some more refactoring

* extern bad arguments

* try to fix style

* improvements and docs

* remove duplicate includes

* fix crash

* make tests more stable by ordering

* rebase once again..

* fix

* make ci happy?

* fix rebase issues

* docs

* rebase, but prolly needs to be improved

* refactor out from nasty inheritance to static methods

* fix style

* work around optional

* refactor & integrate some changes

* update column_type

* add tests by dencrane

* set utc

* fix ref file

* fix tests

* use MergeTree instead of SummingMergeTree

* mark MergeTreeDataPart::getBlock as const

* address a few comments

* compute module function name size at compile time

* simplify branching in getPartitionAstFieldsCount

* remove column_indexes argument

* merge getBlock with buildBlock

* add some const specifiers

* small adjustments

* remove no longer needed isNull check

* use std::min and max to update global min max idx

* add some assertions

* forward declare some symbols

* fix grammar

* forward decl

* try to fix build..

* remove IFunction forward decl

* Revert "use std::min and max to update global min max idx"

This reverts commit b2fe79dda7.

* Revert "remove no longer needed isNull check"

This reverts commit 129db2610f.

* Revert "Revert "remove no longer needed isNull check""

This reverts commit 9416087dd8.

* Revert "Revert "use std::min and max to update global min max idx""

This reverts commit 20246d4416.

* remove some comments

* partial use of MonotonicityCheckMatcher

* ranges

* remove KeyDescriptionMonotonicityChecker

* remove duplication of applyfunction

* move functions to anonymous namespace

* move functions to cpp

* Relax partition compatibility requirements by accepting subset, add tests from partitioned to unpartitioned

* updte reference file

* Support for partition by a, b, c to partition by a, b

* refactoring part 1

* refactoring part 2, use hyperrectangle, still not complete

* refactoring part 3, build hyperrectangle with intersection of source & destination min max columns

* Support attaching to table with partition expression of multiple expressions

* add tests

* rename method

* remove some code duplication

* draft impl for replicatedmergetree, need to dive deeper

* ship ref file

* fix impl for replicatedmergetree..

* forbid attach empty partition replicatedmergetree

* Add replicated merge tree integration tests

* add test missing files

* fix black

* do not check for monotonicity of empty partition

* add empty tests & fix replicated

* remove no longer needed buildBlockWithMinMaxINdexes

* remove column logic in buildHyperrectangle

* simplify implementation by using existing methods

* further simplify implementation

* move all MergeTreeDataPartClone private methods to .cpp file

* decrease decomposition

* use different namespaces

* reduce code duplication

* fix style

* address a few comments

* add chassert to assert arguments size on MonotonicityCheckVisitor

* remove deleteMinMaxFiles method

* remove useless checks from sanitycheck

* add tests for attach partition (not id)

* Remove sanityCheckASTPartition and bring back conditional getPartitionIDFromQuery

* remove empty block comment

* small fixes

* fix formatting

* add missing include

* remove duplicate iuncludes

* trigger ci

* reduce some code duplication

* use updated partition id on replicatedmergetree

* fix build

* fix build

* small refactor

* do not use insert increment on fetch part

* remove duplicate includes

* add one more integ test

* black

* black

* rely on partition exp instead of partition id on replicated part fetch to decide if it is a different partition exp

* add one more integ test

* add order by clause

* fix black

---------

Co-authored-by: Alexey Milovidov <milovidov@clickhouse.com>
 2024-01-22 21:00:04 +01:00
kssenii
1aa8e62bb7 Fix 2024-01-22 18:26:44 +01:00
Nikolai Kochetov
65c5ea31a8
Revert "Revert "Allow parallel replicas for JOIN with analyzer [part 1]."" 2024-01-22 17:09:34 +01:00
Dmitry Novik
4e5249275e
Do not resolve remote table id on initiator 2024-01-22 16:45:25 +01:00
Smita Kulkarni
2e7ce5b0e2 Updated settings ptr and fetching of client from Disk & ObjectStorage 2024-01-22 16:24:43 +01:00
Anton Popov
e1e7a9cd45
Merge branch 'master' into better-read-buffers 2024-01-22 15:50:48 +01:00
Alexander Tokmakov
c2202ff347
Skip unavailable replicas in parallel distributed insert select (#58931) 
* Update StorageDistributed.cpp

* Update StorageDistributed.cpp

* Update StorageDistributed.cpp

* add a test

* Update 01099_parallel_distributed_insert_select.sql
 2024-01-22 15:34:46 +01:00
mochi
0606a77267
Update src/Storages/MergeTree/MutateTask.cpp 
Co-authored-by: Dmitry Novik <mrnovikd@gmail.com>
 2024-01-22 22:02:50 +08:00
Alexander Tokmakov
4315d44517
Revert "Allow parallel replicas for JOIN with analyzer [part 1]." 2024-01-22 13:30:35 +01:00
Kseniia Sumarokova
aec76d6653
Merge pull request #58877 from ClickHouse/multiple-read-file-log-storage-in-mv 
Multiple read file log storage in mv
 2024-01-22 12:02:41 +01:00
MochiXu
02b178cc9c fix drop inverted index 2024-01-22 11:10:58 +08:00
Vitaly Baranov
01aafda27b
Merge pull request #58804 from vitlibar/allow-kill-query-for-backups 
Allow KILL QUERY for backups
 2024-01-20 23:48:18 +01:00
Igor Nikonov
4be068c73b
Merge pull request #58993 from ClickHouse/revert-58992-revert_flaky 
Revive: Parallel replicas custom key: skip unavailable replicas
 2024-01-20 19:44:56 +01:00
robot-clickhouse
273a06856a
Merge pull request #59025 from ClickHouse/cpu-scheduler 
Split resource scheduler off `IO/` into `Common/Scheduler/`
 2024-01-20 15:43:50 +01:00
Kseniia Sumarokova
5d67e26fc4
Merge pull request #58869 from ClickHouse/force-check-for-cached-custom-disk 
Fix for custom cached disk creation
 2024-01-20 14:47:36 +01:00
Kseniia Sumarokova
689e617c12
Merge pull request #58963 from ClickHouse/follow-up-disk-configuration 
Follow up to #58357
 2024-01-20 14:41:34 +01:00
serxa
824deab761 split resource scheduler off IO/ into Common/Scheduler/ 2024-01-20 12:06:26 +00:00
Igor Nikonov
59025b2d3a Merge remote-tracking branch 'origin/master' into revert-58992-revert_flaky 2024-01-19 23:38:24 +00:00
Kruglov Pavel
20379d34d5
Merge branch 'master' into variant-data-type 2024-01-19 23:17:42 +01:00
Michael Kolupaev
fd361273f0 Fix StorageURL forgetting headers on server restart 2024-01-19 11:35:12 -08:00
Alexey Milovidov
decdb62cef
Merge pull request #58800 from ZhiguoZh/20240115-combine-filter-bmi2 
Optimize DB::combineFilters with BMI2/SSE intrinsics
 2024-01-19 16:50:04 +01:00
Dmitry Novik
728276ab3f Fix crash when indexHint() is used 2024-01-19 14:59:33 +00:00
zhongyuankai
cbb51659f6
Merge branch 'master' into limit_materialized_view_size 2024-01-19 22:00:12 +08:00
Igor Nikonov
131f71b50a
Revert "Revert flaky" 2024-01-19 13:51:30 +01:00
Raúl Marín
477489a05d Revert "Merge pull request #57235 from ClickHouse/pr-custom-key-failover" 
This reverts commit 8ba9b4a7ef, reversing
changes made to 0f39245cb8.
 2024-01-19 12:46:05 +00:00
kssenii
2f34ef6ff8 Follow up 2024-01-19 11:19:59 +01:00
kssenii
48e489682b Merge remote-tracking branch 'origin/master' into broken-projections-better-handling 2024-01-19 10:49:27 +01:00
Kseniia Sumarokova
1c74e7fa08
Merge pull request #58902 from ClickHouse/fs-cache-refactorings 
fs cache: minor refactoring
 2024-01-19 10:25:45 +01:00
Robert Schulze
1bfeee1954
Some fixups 2024-01-18 20:49:50 +00:00
Robert Schulze
15700592f7
Merge remote-tracking branch 'ClickHouse/master' into column_level_compress_block 2024-01-18 19:12:57 +00:00
Kruglov Pavel
5444cde408
Merge branch 'master' into variant-data-type 2024-01-18 18:31:27 +01:00
avogar
f202d71371 Make alter with variant espansion as no-op, add tests for alters 2024-01-18 17:29:36 +00:00
Dmitry Novik
ba2dad98bb
Merge pull request #58913 from ClickHouse/fix-style 
Style fix
 2024-01-18 14:24:39 +01:00
Nikolai Kochetov
9db8b50753
Merge pull request #58838 from ClickHouse/allow-parallel-replicas-for-join-with-analyzer 
Allow parallel replicas for JOIN with analyzer [part 1].
 2024-01-18 14:01:08 +01:00
János Benjamin Antal
f34d1ae647 Use WithContext 2024-01-18 11:47:59 +00:00
Smita Kulkarni
e71977b724 Merge branch 'master' into Azure_backup 2024-01-18 08:29:01 +01:00
Duc Canh Le
22afcd46f4 add ALTER COLUMN MODIFY/RESET SETTING 
Signed-off-by: Duc Canh Le <duccanh.le@ahrefs.com>
 2024-01-18 05:17:07 +00:00
Alexander Gololobov
0a42dbafe8 Add attributes 2024-01-17 17:46:15 +01:00
Anton Popov
72b0fed4ef
Merge pull request #58729 from CurtizJ/nested-size0 
Fix reading of offsets subcolumn (`size0`) from `Nested`
 2024-01-17 17:16:27 +01:00
Alexander Gololobov
7dda18c209 Spans in mergeBlock and finalizeImpl 2024-01-17 16:07:53 +01:00
Dmitry Novik
3a1c3638e8 Style fix 2024-01-17 15:04:50 +00:00
Anton Popov
245e1cbd5c Merge remote-tracking branch 'upstream/master' into better-read-buffers 2024-01-17 13:46:48 +00:00
kssenii
c905937ef6 Refactoring 2024-01-17 13:22:59 +01:00
Dmitry Novik
a5e0be484a
Merge pull request #58875 from ClickHouse/fix-indexhint-rpn-builder 
Fix RPN construction for indexHint
 2024-01-17 12:24:28 +01:00
Smita Kulkarni
8d88f4cf87 Update setting is_parameterized_view & settings columns for view 2024-01-17 09:42:53 +01:00
Anton Popov
299293c088 better interface in read buffers 2024-01-16 23:26:20 +00:00