thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-14 18:32:29 +00:00
Code Issues Packages Projects Releases Wiki Activity
134,975 Commits 9,328 Branches 1,783 Tags 2.1 GiB
9361946d15
Commit Graph

16233 Commits

Author SHA1 Message Date
Alexander Gololobov
ca4f46ab41 Check block size in bytes instead of rows 2024-02-06 14:27:15 +01:00
Alexander Gololobov
a26e8b3a9e Fix for filters w/o path column 2024-02-05 22:48:48 +01:00
Vitaly Baranov
86ac7b14d0 Fix Distributed table engine with a constant sharding key. 2024-02-05 21:56:29 +01:00
Igor Nikonov
dab078f7d2 Profile events 2024-02-05 20:21:48 +00:00
Nikolai Kochetov
6b06fcf7fc Fix more tests. 2024-02-05 17:54:15 +00:00
Nikolai Kochetov
15bf263153 Support non global in mode. 2024-02-05 17:05:21 +00:00
Nikolai Kochetov
8692e8f752 Add settings to squash external table blocks. 2024-02-05 17:05:21 +00:00
Nikolai Kochetov
1892318e46 Squash temporary tables. 2024-02-05 17:05:20 +00:00
Nikolai Kochetov
b60228af3f Allow to send a chain of subqueries for parallel replicas with analyzer. 2024-02-05 17:05:20 +00:00
Kseniia Sumarokova
83515e3656
Merge pull request #59543 from nickolaj-jepsen/master 
Add support for NATS credentials file
 2024-02-05 12:20:26 +01:00
Nikita Mikhaylov
6ba35b5459
Make comments for system tables also available in local mode and enforce comments not to be empty (#59493) 
Co-authored-by: Alexey Milovidov <milovidov@clickhouse.com>
 2024-02-05 11:17:58 +01:00
alesapin
db046c1bce
Merge pull request #59517 from ClickHouse/remove_scarry_message 
Remove a scary message if an error is retryable
 2024-02-04 17:43:06 +01:00
Smita Kulkarni
b0994c5fa7 Addressed comments, added test for named collection 2024-02-04 11:28:20 +01:00
Alexander Gololobov
82caaa744a Produce stream of chunks instead of accumulating the whole result 2024-02-03 16:08:41 +01:00
Nickolaj Jepsen
e822ba3a47 Add support for NATS credentials file 2024-02-02 23:34:01 +01:00
Dmitry Novik
05a7d22c24 WIP on calculate aliases only once 2024-02-02 13:45:04 +00:00
Alexander Sapin
d5e3b71590 Remove scarry message if error is retryable 2024-02-02 14:07:37 +01:00
Raúl Marín
87d493533c
Revert "Poco Logger small refactoring" 2024-02-02 12:10:05 +01:00
Alexander Gololobov
095648d7bc
Merge pull request #59388 from ClickHouse/system_zookeeper_retries 
Retry disconnects and expired sessions when reading system.zookeeper
 2024-02-02 11:52:15 +01:00
Alexey Milovidov
15467f2efb
Merge pull request #59451 from ClickHouse/remove-unused-headers 
CLion says these headers are unused
 2024-02-01 21:47:17 +01:00
Nikita Taranov
d73abc17f5 better 2024-02-01 21:42:22 +01:00
Nikita Taranov
698d00dbfe add to MutateTask 2024-02-01 20:43:43 +01:00
Nikita Taranov
76f2ae08b5 collect stats build time 2024-02-01 18:16:09 +01:00
Nikita Taranov
4795149863 print stats once 2024-02-01 17:16:42 +01:00
Kseniia Sumarokova
e029021696
Merge pull request #59446 from ClickHouse/s3-queue-allow-to-add-start-point 
Allow to define a starting point for s3queue ordered mode at creation
 2024-02-01 16:46:24 +01:00
Alexey Milovidov
f52b08070f Merge branch 'master' into remove-unused-headers 2024-02-01 13:42:37 +01:00
kssenii
3a16427e00 Fix test 2024-02-01 12:35:01 +01:00
Maksim Kita
ecbea6f625 Updated implementation 2024-02-01 12:51:50 +03:00
Kseniia Sumarokova
b3418c506e
Update src/Storages/S3Queue/S3QueueSettings.h 
Co-authored-by: Antonio Andelic <antonio2368@users.noreply.github.com>
 2024-02-01 10:28:33 +01:00
Dmitry Novik
5b630ed745
Merge pull request #59399 from azat/build/format-fwd-decl 
Forward declaration for PeekableReadBuffer
 2024-02-01 09:15:07 +01:00
Alexey Milovidov
33f6940214 CLion says these headers are unused 2024-02-01 03:50:09 +01:00
kssenii
361b2f107b Allow to define a starting point for s3queue ordered mode 2024-01-31 21:28:18 +01:00
Alexander Gololobov
5947e5b5a8 Use retry settings and fault injection 2024-01-31 16:48:46 +01:00
Dmitry Novik
572d47acc7 Remove logging 2024-01-31 14:43:06 +00:00
Dmitry Novik
0d21004218 WIP on StorageMerge 2024-01-31 13:50:15 +00:00
Kseniia Sumarokova
b20567a055
Merge pull request #59167 from ClickHouse/s3-queue-parallelize-ordered-mode 
S3Queue: allow parallel & disrtibuted processing for ordered mode
 2024-01-31 11:20:41 +01:00
Kseniia Sumarokova
caaafbcedb
Merge pull request #56864 from ClickHouse/broken-projections-better-handling 
Do not consider data part as broken if projection is broken
 2024-01-31 11:17:15 +01:00
Dmitry Novik
ac17ad1e66
Merge pull request #59073 from ClickHouse/fix-analyzer-resolve-distr-id 
Analyzer: Do not resolve remote table id on initiator
 2024-01-31 09:08:30 +01:00
Nikita Taranov
1be8b61c5b add test 2024-01-30 21:52:32 +01:00
Dmitry Novik
722f3db738 Merge remote-tracking branch 'origin/master' into storage-merge-aliases-analyzer 2024-01-30 19:50:01 +00:00
Nikita Taranov
17ab2674f4 impl 2024-01-30 20:35:10 +01:00
Azat Khuzhin
010f1c6cb7 Forward declaration for PeekableReadBuffer 
ReadHelpers.h is very common header and is the root cause of "recompile
everything".

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-30 19:24:19 +01:00
Alexander Gololobov
c348c4e828 Move ZooKeeperRetries.h to Common 2024-01-30 19:02:17 +01:00
Alexey Milovidov
4ddba907c8 Update autogenerated version to 24.2.1.1 and contributors 2024-01-30 18:49:28 +01:00
Alexander Gololobov
82c06ca294 Use ZooKeeperRetriesControl 2024-01-30 18:31:37 +01:00
Alexander Gololobov
023b8cbd53 Retry disconnects and expired sessions 2024-01-30 17:47:11 +01:00
kssenii
145f6a31cb Fix 2024-01-30 16:29:40 +01:00
Kruglov Pavel
3105960fd8
Merge pull request #59352 from Avogar/fix-bad-attached-rabbitmq 
Fix lazy initialization in RabbitMQ
 2024-01-30 12:52:07 +01:00
Kseniia Sumarokova
e16aaa8d81
Merge branch 'master' into broken-projections-better-handling 2024-01-30 10:54:22 +01:00
Dmitry Novik
5a34641e91 Merge remote-tracking branch 'origin/master' into fix-analyzer-resolve-distr-id 2024-01-30 07:17:15 +00:00
Alexey Milovidov
35ea956255
Merge pull request #59295 from ClickHouse/fix_block_number 
Fix `_block_number` column with non-default serializations and mutation
 2024-01-30 04:07:09 +01:00
Alexey Milovidov
ac90637170
Merge pull request #59122 from arthurpassos/bring_back_attach_partition_pr 
Revert revert attach partition PR
 2024-01-30 04:02:31 +01:00
Alexey Milovidov
de341a6d6e
Merge pull request #59351 from ClickHouse/fix-non-redy-set-system-tables 
Fix not-ready set for system.tables
 2024-01-30 03:59:18 +01:00
Alexey Milovidov
50532c485f
Merge pull request #59340 from CurtizJ/lower-memory-usage-vertical-merges 
Lower memory usage in vertical merges
 2024-01-30 03:42:28 +01:00
avogar
3a5ba56c36 Fix lazy initialization in RabbitMQ, fix possible deadlock on insert into unitialized queue engine 2024-01-29 20:09:09 +00:00
Robert Schulze
2cfcaf8ff4
Merge pull request #59040 from MochiXu/fix_remove_inverted_index_files 
`DROP INDEX` of inverted index now removes all relevant files from persistence
 2024-01-29 20:39:56 +01:00
kssenii
df7d3b8a40 Allocate current shard number from keeper 2024-01-29 19:36:16 +01:00
Arthur Passos
15d04d0a78
Merge branch 'master' into bring_back_attach_partition_pr 2024-01-29 15:16:49 -03:00
Arthur Passos
516daadeef add static storage assert in datapartcloner 2024-01-29 15:16:00 -03:00
Nikolai Kochetov
d0143380d7 Fix not-ready set for system.tables 2024-01-29 17:53:59 +00:00
Nikolai Kochetov
5b2648929b
Merge pull request #59273 from kitaisreal/atomic-logger 
Added AtomicLogger
 2024-01-29 16:49:18 +01:00
avogar
377937d415 Merge branch 'master' of github.com:ClickHouse/ClickHouse into auto-format-detection 2024-01-29 15:45:18 +00:00
Anton Popov
d3cdc8826e lower memory usage in vertical merges 2024-01-29 14:06:46 +00:00
kssenii
d0ea5392db Fix storing iterated keys 2024-01-29 12:36:32 +01:00
kssenii
ef252d8376 Validate settings with keeper, add a test 2024-01-29 12:36:00 +01:00
Kruglov Pavel
6858d2f4ca
Merge pull request #58047 from Avogar/variant-data-type 
Implement Variant data type
 2024-01-29 11:36:08 +01:00
Smita Kulkarni
3183026521 Merge branch 'master' into Azure_backup 2024-01-29 11:08:13 +01:00
Kseniia Sumarokova
a29ab8f29c
Merge pull request #59170 from azat/disks/s3-plain-improvements 
Small improvements for tables on write-once (s3_plain)/read-only (web) disks
 2024-01-29 10:44:37 +01:00
Kseniia Sumarokova
0f88da5fd4
Merge pull request #59275 from ClickHouse/Avogar-patch-1 
Fix abort in iceberg metadata on bad file paths
 2024-01-29 10:35:34 +01:00
alesapin
b864b122fa Merge branch 'master' into fix_block_number 2024-01-29 10:19:43 +01:00
Alexey Milovidov
2cb2bcfbc3
Merge pull request #59315 from ClickHouse/support-backups-for-compressed-memory-tables 
Support backups for compressed in-memory tables
 2024-01-29 01:50:41 +01:00
Alexey Milovidov
aec3f28ccb Support backups for compressed in-memory tables 2024-01-28 23:06:50 +01:00
Arthur Passos
0bbfef884c
Merge branch 'ClickHouse:master' into bring_back_attach_partition_pr 2024-01-28 17:43:38 -03:00
Anton Popov
ef133bfd47 Merge remote-tracking branch 'upstream/master' into lower-memory-usage 2024-01-28 15:07:44 +00:00
avogar
92db86b829 Merge branch 'master' of github.com:ClickHouse/ClickHouse into variant-data-type 2024-01-28 14:33:49 +00:00
Maksim Kita
07750ee949 Added AtomicLogger 2024-01-28 13:55:44 +03:00
alesapin
7022ca9ece Fix _block_number column with non-default serializations and mutation 2024-01-27 20:24:03 +01:00
Anton Popov
9d3b4e5308 lower memory usage 2024-01-27 16:40:11 +00:00
Azat Khuzhin
d94547d338 Use fresh ZooKeeper client on DROP (to have higher chances on success) 
In case of DROP the client can be expired, and even though
StorageReplicatedMergeTree::drop() uses getZooKeeperIfTableShutDown(),
which creates new client if current is expired, it is not enough, since
current_zookeeper (cached client) will be used implicitly from
dropAllData().

This had been found by stress tests, that leads to DROP query hang [1].

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/59255/94eb33ef27a9ab7c4a99af40772ea287e67efcbf/stress_test__tsan_.html

<details>

<summary>stacktrace</summary>

    2024.01.26 17:41:41.138577 [ 3319 ] {} <Error> DatabaseCatalog: Cannot drop table test_vzfk1xoc.alter_table1 (455a674c-161d-44d8-abc2-fd205bad1116). Will retry later.: Code: 999. Coordination::Exception: Session expired. (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

    0. ./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000001e06a5e3 in /usr/bin/clickhouse
    1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f697c74 in /usr/bin/clickhouse
    2. ./src/Common/Exception.h:0: Coordination::Exception::Exception<char const (&) [16]>(char const (&) [16], Coordination::Error) @ 0x000000001adc298b in /usr/bin/clickhouse
    3. ./src/Common/ZooKeeper/IKeeper.h:0: Coordination::ZooKeeper::pushRequest(Coordination::ZooKeeper::RequestInfo&&) @ 0x000000001adfb418 in /usr/bin/clickhouse
    4. ./build_docker/./src/Common/ZooKeeper/ZooKeeperImpl.cpp:1343: Coordination::ZooKeeper::get(String const&, std::function<void (Coordination::GetResponse const&)>, std::shared_ptr<std::function<void (Coordination::WatchResponse const&)>>) @ 0x000000001adfd8e5 in /usr/bin/clickhouse
    5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x000000001ad890bd in /usr/bin/clickhouse
    6. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x000000001ad88a36 in /usr/bin/clickhouse
    7. ./build_docker/./src/Common/ZooKeeper/ZooKeeper.cpp:580: zkutil::ZooKeeper::tryGetWatch(String const&, String&, Coordination::Stat*, std::function<void (Coordination::WatchResponse const&)>, Coordination::Error*) @ 0x000000001ad898bf in /usr/bin/clickhouse
    8. ./build_docker/./src/Common/ZooKeeper/ZooKeeper.cpp:570: zkutil::ZooKeeper::tryGet(String const&, String&, Coordination::Stat*, std::shared_ptr<Poco::Event> const&, Coordination::Error*) @ 0x000000001ad89554 in /usr/bin/clickhouse
    9. ./build_docker/./src/Common/ZooKeeper/ZooKeeperWithFaultInjection.cpp:0: DB::ZooKeeperWithFaultInjection::tryGet(String const&, String&, Coordination::Stat*, std::shared_ptr<Poco::Event> const&, Coordination::Error*) @ 0x000000001ae110dc in /usr/bin/clickhouse
    10. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:701: DB::StorageReplicatedMergeTree::unlockSharedDataByID(String, String const&, DB::MergeTreePartInfo const&, String const&, String const&, std::shared_ptr<DB::ZooKeeperWithFaultInjection> const&, DB::MergeTreeSettings const&, std::shared_ptr<Poco::Logger>, String const&, StrongTypedef<unsigned int, DB::MergeTreeDataFormatVersionTag>) @ 0x000000001975527e in /usr/bin/clickhouse
    11. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::removeSharedDetachedPart(std::shared_ptr<DB::IDisk>, String const&, String const&, String const&, String const&, String const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<zkutil::ZooKeeper> const&) @ 0x000000001976c980 in /usr/bin/clickhouse
    12. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:10171: DB::StorageReplicatedMergeTree::removeDetachedPart(std::shared_ptr<DB::IDisk>, String const&, String const&) @ 0x000000001976bdc6 in /usr/bin/clickhouse
    13. ./build_docker/./src/Storages/MergeTree/MergeTreeData.cpp:2798: DB::MergeTreeData::dropAllData() @ 0x0000000019c643c8 in /usr/bin/clickhouse
    14. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:1117: DB::StorageReplicatedMergeTree::drop() @ 0x0000000019664e0a in /usr/bin/clickhouse
    15. ./build_docker/./src/Interpreters/DatabaseCatalog.cpp:0: DB::DatabaseCatalog::dropTableFinally(DB::DatabaseCatalog::TableMarkedAsDropped const&) @ 0x0000000017bf1fac in /usr/bin/clickhouse
    16. ./build_docker/./src/Interpreters/DatabaseCatalog.cpp:0: DB::DatabaseCatalog::dropTableDataTask() @ 0x0000000017bf175e in /usr/bin/clickhouse
    17. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x0000000017bf94c2 in /usr/bin/clickhouse
    18. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000170bd0b8 in /usr/bin/clickhouse
    19. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:701: DB::BackgroundSchedulePool::threadFunction() @ 0x00000000170c06ce in /usr/bin/clickhouse
    20. ./build_docker/./src/Core/BackgroundSchedulePool.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000170c0feb in /usr/bin/clickhouse
    21. ./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f7c6a2f in /usr/bin/clickhouse
    22. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f7cc8f2 in /usr/bin/clickhouse
    23. ? @ 0x000000000733904f in /usr/bin/clickhouse
    24. ? @ 0x00007f28b2319ac3
    25. ? @ 0x00007f28b23ab850

</details>

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-27 16:04:18 +01:00
kssenii
0a21e6aa5d Fix 2024-01-26 19:07:30 +01:00
avogar
5833641fa5 Merge branch 'master' of github.com:ClickHouse/ClickHouse into variant-data-type 2024-01-26 16:54:02 +00:00
Kruglov Pavel
8fcedddcda
Fix abort in iceberg metadata on bad file paths 2024-01-26 17:46:49 +01:00
Azat Khuzhin
90ab986be6 Revert "Prohibit CREATE on static storages (ATTACH should be used instead)" 
CREATE is used by Replicated database even for ATTACH, and anyway ATTACH
creates format_version.txt as well (without previuos patch).

This reverts commit 323f1f320ebe7e588d443abf78efa2c60193e7a9.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-26 16:47:36 +01:00
Azat Khuzhin
7ba31bf55c Do not create format_version.txt on write-once storages 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-26 16:47:36 +01:00
Azat Khuzhin
66a40bda0a Revert "Prohibit DROP from static storages" 
It is actually OK to DROP at least from read-only disk, as for the
write-once it should be OK as well, since CREATE is prohibited anyway,
so there should be no leftovers

CI: https://s3.amazonaws.com/clickhouse-test-reports/59170/c82050d1925439f0ede2b32acb5b1b8df4acae5d/stress_test__asan_.html
CI: https://s3.amazonaws.com/clickhouse-test-reports/59170/c82050d1925439f0ede2b32acb5b1b8df4acae5d/integration_tests__asan__[3_4].html

This reverts commit 082e62712c5e2f4c48d1abc6141ccfd1249d26b4.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-26 16:47:35 +01:00
Azat Khuzhin
aafc149c1e Prohibit CREATE on static storages (ATTACH should be used instead) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-26 16:47:35 +01:00
Azat Khuzhin
1ee723d50d Prohibit DROP from static storages 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-26 16:47:35 +01:00
Azat Khuzhin
f3b4f28f0d Disable mutable operations into {Replicated,}MergeTree with static storage 
Mutable operations includes:
- INSERT
- OPTIMIZE
- parts manipulation
- ...

But does not include DROP, since you need an ability to DROP it.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-26 16:47:35 +01:00
Azat Khuzhin
6751e01cc3 Do not drop parts for static (ro/write-once) storages for Replicated engine 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-26 16:47:35 +01:00
kssenii
5cd757dd9a Merge remote-tracking branch 'origin/master' into s3-queue-parallelize-ordered-mode 2024-01-26 16:33:22 +01:00
Kseniia Sumarokova
99653fc778
Merge branch 'master' into broken-projections-better-handling 2024-01-26 16:27:23 +01:00
kssenii
ee15b6e646 Fix 2024-01-26 16:26:59 +01:00
Kruglov Pavel
946c4e0495
Merge branch 'master' into auto-format-detection 2024-01-26 15:51:35 +01:00
Kruglov Pavel
692d37306e
Fix build 2024-01-26 15:50:16 +01:00
Arthur Passos
ce6877cbb6
Merge branch 'ClickHouse:master' into bring_back_attach_partition_pr 2024-01-26 10:57:11 -03:00
Kruglov Pavel
46a6b84a5a
Merge branch 'master' into auto-format-detection 2024-01-25 22:11:07 +01:00
Maksim Kita
5bb734a4bb ActionsDAG buildFilterActionsDAG refactoring 2024-01-25 18:24:14 +03:00
Maksim Kita
2a327107b6 Updated implementation 2024-01-25 14:31:49 +03:00
Maksim Kita
42029f42e7 Table CREATE DROP Poco::Logger memory leak fix 2024-01-25 14:31:18 +03:00
Kseniia Sumarokova
951c874aa9
Merge pull request #59080 from ClickHouse/data-lakes-survive-restart 
Allow server to start with broken data lake table and fix itself lazily
 2024-01-25 11:50:16 +01:00
Nikita Taranov
1e3f6894a3
Merge pull request #59089 from nickitat/sort_res_ranges_before_sending_to_replica 
Coordinator returns ranges for reading in sorted order
 2024-01-24 21:04:53 +01:00
avogar
11f1ea50d7 Fix tests 2024-01-24 17:55:31 +00:00
kssenii
5ecf36f275 Fix style check 2024-01-24 17:19:52 +01:00
Dmitry Novik
95e2a3d6da
Merge pull request #58911 from ClickHouse/fix-index-hint-crash 
Fix crash when indexHint() is used
 2024-01-24 17:16:22 +01:00
kssenii
7f8f379d7f Parallel & disrtibuted processing for ordered mode 2024-01-24 16:32:15 +01:00
Kruglov Pavel
89bafbcda6
Merge pull request #59133 from Avogar/allow-to-ignore-schema-evolution-in-iceberg 
Allow to ignore schema evolution in Iceberg table engine under a setting
 2024-01-24 13:19:09 +01:00
Smita Kulkarni
4d92eba316 Merge branch 'master' into Azure_backup 2024-01-24 11:13:57 +01:00
avogar
f05174e441 Fix style 2024-01-23 18:59:40 +00:00
avogar
617cc514b7 Try to detect file format automatically during schema inference if it's unknown 2024-01-23 18:59:39 +00:00
Arthur Passos
dba792dff0
Remove duplicate includ 2024-01-23 15:03:31 -03:00
Arthur Passos
e6b17377d0
Merge branch 'master' into bring_back_attach_partition_pr 2024-01-23 14:52:47 -03:00
avogar
be3b5dc45f Allow to ignore schema evolution in Iceberg table engine under a setting 2024-01-23 17:45:41 +00:00
Nikolai Kochetov
d64b491b45 Fixing build 2024-01-23 17:00:21 +00:00
Nikolai Kochetov
a26f8f5425
Merge pull request #59076 from ClickHouse/revert-59059-revert-58838-allow-parallel-replicas-for-join-with-analyzer 
Revert "Revert "Allow parallel replicas for JOIN with analyzer [part 1].""
 2024-01-23 17:01:19 +01:00
Arthur Passos
5179891aef remove static log 2024-01-23 11:43:08 -03:00
Arthur Passos
799a94081b Revert "Revert "Allow to attach partition from table with different partition expression when destination partition expression doesn't re-partition"" 
This reverts commit a1c83e2f51.
 2024-01-23 11:40:42 -03:00
Robert Schulze
a1c83e2f51
Revert "Allow to attach partition from table with different partition expression when destination partition expression doesn't re-partition" 2024-01-23 15:26:04 +01:00
Anton Popov
4292ee7d51
Merge pull request #58886 from CurtizJ/better-read-buffers 
Less error prone interface of read buffers
 2024-01-23 14:59:27 +01:00
Alexander Gololobov
9a0e5d0aac
Merge pull request #58914 from ClickHouse/ot_spans_in_insert 
Opentelemetry spans to analyze CPU and S3 bottlenecks on inserts
 2024-01-23 13:53:13 +01:00
Nikolai Kochetov
fcbab407b8
Merge branch 'master' into revert-59059-revert-58838-allow-parallel-replicas-for-join-with-analyzer 2024-01-23 12:21:44 +01:00
kssenii
bca73d4b0a Fix 2024-01-23 11:51:21 +01:00
kssenii
b76630c955 Merge remote-tracking branch 'origin/master' into broken-projections-better-handling 2024-01-23 11:34:58 +01:00
Robert Schulze
e67076ea2b
Merge pull request #55201 from canhld94/column_level_compress_block 
Column-level compression block sizes
 2024-01-23 11:22:03 +01:00
Robert Schulze
bfec324b28
Some fixups + test 2024-01-23 10:17:16 +00:00
Robert Schulze
81543c0a46
Merge remote-tracking branch 'rschu1ze/master' into fix_remove_inverted_index_files 2024-01-23 09:45:09 +00:00
Nikita Taranov
09e24ed6c5
Merge pull request #58068 from zhongyuankai/limit_materialized_view_size 
A limit on the number of materialized views attached to a table.
 2024-01-22 22:50:56 +01:00
Nikita Taranov
b161921825 impl 2024-01-22 22:33:23 +01:00
Max K
4932b1cad9
Merge branch 'master' into better-read-buffers 2024-01-22 21:52:55 +01:00
Arthur Passos
24b8bbe9fa
Allow to attach partition from table with different partition expression when destination partition expression doesn't re-partition (#39507) 
* temp commit

* temp commit

* draft impl for feedback

* fix weird style changes

* fix weird style changes

* fix weird style changes

* fix weird style changes

* fix weird style changes

* aa

* aa

* Add integ tests and remove partition key restriction

* fix small incosistency in partition id

* style fix

* style fix

* style fix

* use existing DataPartStorageBuilder instead of new one

* Refactor part clone to make it more readable and maintainable

* Add MergeTreeDataPartCloner docs

* define ErrorCodes::BAD_ARGUMENTS

* Rebase

* camel case methods

* address some comments

* yet another rebase?

* Move from integ tests to stateless tests

* address more comments

* add finalize on min_max_idx files

* Add sync option to DistinctPartitionExpCloner

* just a temp test

* revert temp change

* Use echoOn to distinguish test queries

* remove comment

* fix build issue during rebase

* atempt to fix build after rebase

* finally fix build

* clear minmaxidx hyperrectangle before loading it

* Fix error on min_max files deletion where it was being assumed that partition expression contained all columns

* get it to the state it was previously

* add missing include

* getting functional?

* refactoring and renaming

* some more refactoring

* extern bad arguments

* try to fix style

* improvements and docs

* remove duplicate includes

* fix crash

* make tests more stable by ordering

* rebase once again..

* fix

* make ci happy?

* fix rebase issues

* docs

* rebase, but prolly needs to be improved

* refactor out from nasty inheritance to static methods

* fix style

* work around optional

* refactor & integrate some changes

* update column_type

* add tests by dencrane

* set utc

* fix ref file

* fix tests

* use MergeTree instead of SummingMergeTree

* mark MergeTreeDataPart::getBlock as const

* address a few comments

* compute module function name size at compile time

* simplify branching in getPartitionAstFieldsCount

* remove column_indexes argument

* merge getBlock with buildBlock

* add some const specifiers

* small adjustments

* remove no longer needed isNull check

* use std::min and max to update global min max idx

* add some assertions

* forward declare some symbols

* fix grammar

* forward decl

* try to fix build..

* remove IFunction forward decl

* Revert "use std::min and max to update global min max idx"

This reverts commit b2fe79dda7.

* Revert "remove no longer needed isNull check"

This reverts commit 129db2610f.

* Revert "Revert "remove no longer needed isNull check""

This reverts commit 9416087dd8.

* Revert "Revert "use std::min and max to update global min max idx""

This reverts commit 20246d4416.

* remove some comments

* partial use of MonotonicityCheckMatcher

* ranges

* remove KeyDescriptionMonotonicityChecker

* remove duplication of applyfunction

* move functions to anonymous namespace

* move functions to cpp

* Relax partition compatibility requirements by accepting subset, add tests from partitioned to unpartitioned

* updte reference file

* Support for partition by a, b, c to partition by a, b

* refactoring part 1

* refactoring part 2, use hyperrectangle, still not complete

* refactoring part 3, build hyperrectangle with intersection of source & destination min max columns

* Support attaching to table with partition expression of multiple expressions

* add tests

* rename method

* remove some code duplication

* draft impl for replicatedmergetree, need to dive deeper

* ship ref file

* fix impl for replicatedmergetree..

* forbid attach empty partition replicatedmergetree

* Add replicated merge tree integration tests

* add test missing files

* fix black

* do not check for monotonicity of empty partition

* add empty tests & fix replicated

* remove no longer needed buildBlockWithMinMaxINdexes

* remove column logic in buildHyperrectangle

* simplify implementation by using existing methods

* further simplify implementation

* move all MergeTreeDataPartClone private methods to .cpp file

* decrease decomposition

* use different namespaces

* reduce code duplication

* fix style

* address a few comments

* add chassert to assert arguments size on MonotonicityCheckVisitor

* remove deleteMinMaxFiles method

* remove useless checks from sanitycheck

* add tests for attach partition (not id)

* Remove sanityCheckASTPartition and bring back conditional getPartitionIDFromQuery

* remove empty block comment

* small fixes

* fix formatting

* add missing include

* remove duplicate iuncludes

* trigger ci

* reduce some code duplication

* use updated partition id on replicatedmergetree

* fix build

* fix build

* small refactor

* do not use insert increment on fetch part

* remove duplicate includes

* add one more integ test

* black

* black

* rely on partition exp instead of partition id on replicated part fetch to decide if it is a different partition exp

* add one more integ test

* add order by clause

* fix black

---------

Co-authored-by: Alexey Milovidov <milovidov@clickhouse.com>
 2024-01-22 21:00:04 +01:00
kssenii
1aa8e62bb7 Fix 2024-01-22 18:26:44 +01:00
Nikolai Kochetov
65c5ea31a8
Revert "Revert "Allow parallel replicas for JOIN with analyzer [part 1]."" 2024-01-22 17:09:34 +01:00
Dmitry Novik
4e5249275e
Do not resolve remote table id on initiator 2024-01-22 16:45:25 +01:00
Smita Kulkarni
2e7ce5b0e2 Updated settings ptr and fetching of client from Disk & ObjectStorage 2024-01-22 16:24:43 +01:00
Anton Popov
e1e7a9cd45
Merge branch 'master' into better-read-buffers 2024-01-22 15:50:48 +01:00
Alexander Tokmakov
c2202ff347
Skip unavailable replicas in parallel distributed insert select (#58931) 
* Update StorageDistributed.cpp

* Update StorageDistributed.cpp

* Update StorageDistributed.cpp

* add a test

* Update 01099_parallel_distributed_insert_select.sql
 2024-01-22 15:34:46 +01:00
mochi
0606a77267
Update src/Storages/MergeTree/MutateTask.cpp 
Co-authored-by: Dmitry Novik <mrnovikd@gmail.com>
 2024-01-22 22:02:50 +08:00
Alexander Tokmakov
4315d44517
Revert "Allow parallel replicas for JOIN with analyzer [part 1]." 2024-01-22 13:30:35 +01:00
Kseniia Sumarokova
aec76d6653
Merge pull request #58877 from ClickHouse/multiple-read-file-log-storage-in-mv 
Multiple read file log storage in mv
 2024-01-22 12:02:41 +01:00
MochiXu
02b178cc9c fix drop inverted index 2024-01-22 11:10:58 +08:00
Vitaly Baranov
01aafda27b
Merge pull request #58804 from vitlibar/allow-kill-query-for-backups 
Allow KILL QUERY for backups
 2024-01-20 23:48:18 +01:00
Igor Nikonov
4be068c73b
Merge pull request #58993 from ClickHouse/revert-58992-revert_flaky 
Revive: Parallel replicas custom key: skip unavailable replicas
 2024-01-20 19:44:56 +01:00
robot-clickhouse
273a06856a
Merge pull request #59025 from ClickHouse/cpu-scheduler 
Split resource scheduler off `IO/` into `Common/Scheduler/`
 2024-01-20 15:43:50 +01:00
Kseniia Sumarokova
5d67e26fc4
Merge pull request #58869 from ClickHouse/force-check-for-cached-custom-disk 
Fix for custom cached disk creation
 2024-01-20 14:47:36 +01:00
Kseniia Sumarokova
689e617c12
Merge pull request #58963 from ClickHouse/follow-up-disk-configuration 
Follow up to #58357
 2024-01-20 14:41:34 +01:00
serxa
824deab761 split resource scheduler off IO/ into Common/Scheduler/ 2024-01-20 12:06:26 +00:00
Igor Nikonov
59025b2d3a Merge remote-tracking branch 'origin/master' into revert-58992-revert_flaky 2024-01-19 23:38:24 +00:00
Kruglov Pavel
20379d34d5
Merge branch 'master' into variant-data-type 2024-01-19 23:17:42 +01:00
Michael Kolupaev
fd361273f0 Fix StorageURL forgetting headers on server restart 2024-01-19 11:35:12 -08:00
Alexey Milovidov
decdb62cef
Merge pull request #58800 from ZhiguoZh/20240115-combine-filter-bmi2 
Optimize DB::combineFilters with BMI2/SSE intrinsics
 2024-01-19 16:50:04 +01:00
Dmitry Novik
728276ab3f Fix crash when indexHint() is used 2024-01-19 14:59:33 +00:00