thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-27 10:02:01 +00:00
Code Issues Packages Projects Releases Wiki Activity
133,780 Commits 9,196 Branches 1,782 Tags 2.1 GiB
361b2f107b
Commit Graph

133780 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
slvrtrn
eb909a5db4 Reapply add MySQL net write/read settings 2024-01-27 18:36:38 +01:00
Smita Kulkarni
b7a6dbfbb9 Check engine to avoid possible seg fault 2024-01-27 18:22:37 +01:00
Anton Popov
9d3b4e5308 lower memory usage 2024-01-27 16:40:11 +00:00
Peignon Melvyn
cf1df96cc1
Update view.md 
Improved documentation.
 2024-01-27 17:27:00 +01:00
Azat Khuzhin
d94547d338 Use fresh ZooKeeper client on DROP (to have higher chances on success) 
In case of DROP the client can be expired, and even though
StorageReplicatedMergeTree::drop() uses getZooKeeperIfTableShutDown(),
which creates new client if current is expired, it is not enough, since
current_zookeeper (cached client) will be used implicitly from
dropAllData().

This had been found by stress tests, that leads to DROP query hang [1].

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/59255/94eb33ef27a9ab7c4a99af40772ea287e67efcbf/stress_test__tsan_.html

<details>

<summary>stacktrace</summary>

    2024.01.26 17:41:41.138577 [ 3319 ] {} <Error> DatabaseCatalog: Cannot drop table test_vzfk1xoc.alter_table1 (455a674c-161d-44d8-abc2-fd205bad1116). Will retry later.: Code: 999. Coordination::Exception: Session expired. (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below):

    0. ./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000001e06a5e3 in /usr/bin/clickhouse
    1. ./build_docker/./src/Common/Exception.cpp:96: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f697c74 in /usr/bin/clickhouse
    2. ./src/Common/Exception.h:0: Coordination::Exception::Exception<char const (&) [16]>(char const (&) [16], Coordination::Error) @ 0x000000001adc298b in /usr/bin/clickhouse
    3. ./src/Common/ZooKeeper/IKeeper.h:0: Coordination::ZooKeeper::pushRequest(Coordination::ZooKeeper::RequestInfo&&) @ 0x000000001adfb418 in /usr/bin/clickhouse
    4. ./build_docker/./src/Common/ZooKeeper/ZooKeeperImpl.cpp:1343: Coordination::ZooKeeper::get(String const&, std::function<void (Coordination::GetResponse const&)>, std::shared_ptr<std::function<void (Coordination::WatchResponse const&)>>) @ 0x000000001adfd8e5 in /usr/bin/clickhouse
    5. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x000000001ad890bd in /usr/bin/clickhouse
    6. ./contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x000000001ad88a36 in /usr/bin/clickhouse
    7. ./build_docker/./src/Common/ZooKeeper/ZooKeeper.cpp:580: zkutil::ZooKeeper::tryGetWatch(String const&, String&, Coordination::Stat*, std::function<void (Coordination::WatchResponse const&)>, Coordination::Error*) @ 0x000000001ad898bf in /usr/bin/clickhouse
    8. ./build_docker/./src/Common/ZooKeeper/ZooKeeper.cpp:570: zkutil::ZooKeeper::tryGet(String const&, String&, Coordination::Stat*, std::shared_ptr<Poco::Event> const&, Coordination::Error*) @ 0x000000001ad89554 in /usr/bin/clickhouse
    9. ./build_docker/./src/Common/ZooKeeper/ZooKeeperWithFaultInjection.cpp:0: DB::ZooKeeperWithFaultInjection::tryGet(String const&, String&, Coordination::Stat*, std::shared_ptr<Poco::Event> const&, Coordination::Error*) @ 0x000000001ae110dc in /usr/bin/clickhouse
    10. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:701: DB::StorageReplicatedMergeTree::unlockSharedDataByID(String, String const&, DB::MergeTreePartInfo const&, String const&, String const&, std::shared_ptr<DB::ZooKeeperWithFaultInjection> const&, DB::MergeTreeSettings const&, std::shared_ptr<Poco::Logger>, String const&, StrongTypedef<unsigned int, DB::MergeTreeDataFormatVersionTag>) @ 0x000000001975527e in /usr/bin/clickhouse
    11. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:0: DB::StorageReplicatedMergeTree::removeSharedDetachedPart(std::shared_ptr<DB::IDisk>, String const&, String const&, String const&, String const&, String const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<zkutil::ZooKeeper> const&) @ 0x000000001976c980 in /usr/bin/clickhouse
    12. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:10171: DB::StorageReplicatedMergeTree::removeDetachedPart(std::shared_ptr<DB::IDisk>, String const&, String const&) @ 0x000000001976bdc6 in /usr/bin/clickhouse
    13. ./build_docker/./src/Storages/MergeTree/MergeTreeData.cpp:2798: DB::MergeTreeData::dropAllData() @ 0x0000000019c643c8 in /usr/bin/clickhouse
    14. ./build_docker/./src/Storages/StorageReplicatedMergeTree.cpp:1117: DB::StorageReplicatedMergeTree::drop() @ 0x0000000019664e0a in /usr/bin/clickhouse
    15. ./build_docker/./src/Interpreters/DatabaseCatalog.cpp:0: DB::DatabaseCatalog::dropTableFinally(DB::DatabaseCatalog::TableMarkedAsDropped const&) @ 0x0000000017bf1fac in /usr/bin/clickhouse
    16. ./build_docker/./src/Interpreters/DatabaseCatalog.cpp:0: DB::DatabaseCatalog::dropTableDataTask() @ 0x0000000017bf175e in /usr/bin/clickhouse
    17. ./contrib/llvm-project/libcxx/include/__functional/function.h:717: ? @ 0x0000000017bf94c2 in /usr/bin/clickhouse
    18. ./contrib/llvm-project/libcxx/include/__functional/function.h:0: ? @ 0x00000000170bd0b8 in /usr/bin/clickhouse
    19. ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:701: DB::BackgroundSchedulePool::threadFunction() @ 0x00000000170c06ce in /usr/bin/clickhouse
    20. ./build_docker/./src/Core/BackgroundSchedulePool.cpp:0: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000170c0feb in /usr/bin/clickhouse
    21. ./base/base/../base/wide_integer_impl.h:809: ThreadPoolImpl<std::thread>::worker(std::__list_iterator<std::thread, void*>) @ 0x000000000f7c6a2f in /usr/bin/clickhouse
    22. ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000f7cc8f2 in /usr/bin/clickhouse
    23. ? @ 0x000000000733904f in /usr/bin/clickhouse
    24. ? @ 0x00007f28b2319ac3
    25. ? @ 0x00007f28b23ab850

</details>

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-27 16:04:18 +01:00
Azat Khuzhin
71cd1efd8b tests: fix 02980_s3_plain_DROP_TABLE_ReplicatedMergeTree flakiness 
In case of commit failed (likely due to fault injection in tests) part
name will have different name, not all_0_0_0, so replace numbers with X.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-27 15:19:21 +01:00
Alexey Milovidov
661d4a827b
Merge pull request #59272 from azat/osx-rust 
Enable Rust on OSX ARM64
 2024-01-27 14:29:32 +01:00
Maksim Kita
7d48d0da72 Fixed tests 2024-01-27 15:57:49 +03:00
Maksim Kita
0f46200d5a Fixed tests 2024-01-27 15:57:49 +03:00
Maksim Kita
4a10b71252 Updated documentation 2024-01-27 15:57:49 +03:00
Maksim Kita
4ae425461f Added documentation 2024-01-27 15:57:49 +03:00
Maksim Kita
5d62cdd58c Added setting update_insert_deduplication_token_in_dependent_materialized_views 2024-01-27 15:57:48 +03:00
Azat Khuzhin
44e42052b1 Fix perf tests after sumMap starts to filter out -0. 
Before perf tests was relying on the following:

    SELECT sumMap(['foo', 'bar'], [-0., -0.])

    ┌─sumMap(['foo', 'bar'], [-0., -0.])─┐
    │ (['bar','foo'],[-0,-0])            │
    └────────────────────────────────────┘

While it got changed, and now:

    ┌─sumMap(['foo', 'bar'], [-0., -0.])─┐
    │ ([],[])                            │
    └────────────────────────────────────┘

But it works for nan:

    SELECT sumMap(['foo', 'bar'], [nan, nan])

    ┌─sumMap(['foo', 'bar'], [nan, nan])─┐
    │ (['bar','foo'],[nan,nan])          │
    └────────────────────────────────────┘

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-27 13:14:07 +01:00
Kruglov Pavel
875efc54b8
Merge pull request #59283 from ClickHouse/Avogar-patch-2 
Add missing setting optimize_injective_functions_in_group_by to SettingsChangesHistory
 2024-01-27 00:12:10 +01:00
Kruglov Pavel
1e61521dcc
Add missing setting optimize_injective_functions_in_group_by to SettingsChangesHistory 2024-01-26 23:33:04 +01:00
Peignon Melvyn
4273ae4f4f
Update view.md 2024-01-26 21:27:20 +01:00
Kseniia Sumarokova
163a3e44d3
Update run.sh 2024-01-26 21:22:39 +01:00
Kseniia Sumarokova
2c6243f365
Update test.py 2024-01-26 19:52:53 +01:00
robot-ch-test-poll2
930ce11ae4
Merge pull request #59261 from artbeglaryan/patch-3 
Fixed typo in documentation
 2024-01-26 22:41:04 +04:00
kssenii
0a21e6aa5d Fix 2024-01-26 19:07:30 +01:00
Aleksandr Musorin
e6352234c3 Added parsing trailing symbols for type expressions 
```sql
SELECT (1, 'foo')::Tuple(a Int, b String,);
SELECT (1, 'foo')::Tuple(Int, String,);
SELECT (1, (2,'foo'))::Tuple(Int, Tuple(Int, String,),);
````

fix
 2024-01-26 17:56:14 +01:00
avogar
5833641fa5 Merge branch 'master' of github.com:ClickHouse/ClickHouse into variant-data-type 2024-01-26 16:54:02 +00:00
Kruglov Pavel
8fcedddcda
Fix abort in iceberg metadata on bad file paths 2024-01-26 17:46:49 +01:00
vdimir
6795d0207f
Update test_reload_after_fail_in_cache_dictionary for analyzer 2024-01-26 16:37:54 +00:00
Arthur Passos
d8deebaaba
Merge branch 'ClickHouse:master' into bring_back_attach_partition_pr 2024-01-26 13:27:57 -03:00
Azat Khuzhin
50f0a34ecb Enable Rust on OSX 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-26 17:20:00 +01:00
Azat Khuzhin
5227d7c9bc Fix 02475_bson_each_row_format flakiness (due to small parsing block) 
CI: https://s3.amazonaws.com/clickhouse-test-reports/59170/c82050d1925439f0ede2b32acb5b1b8df4acae5d/stateless_tests__release_/run.log
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-26 16:51:37 +01:00
Azat Khuzhin
332924a2b4 tests: adjust 02980_s3_plain_DROP_TABLE tests 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-26 16:47:36 +01:00
Azat Khuzhin
90ab986be6 Revert "Prohibit CREATE on static storages (ATTACH should be used instead)" 
CREATE is used by Replicated database even for ATTACH, and anyway ATTACH
creates format_version.txt as well (without previuos patch).

This reverts commit 323f1f320ebe7e588d443abf78efa2c60193e7a9.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-26 16:47:36 +01:00
Azat Khuzhin
7ba31bf55c Do not create format_version.txt on write-once storages 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-26 16:47:36 +01:00
Azat Khuzhin
7cc7fb892c tests/test_disk_over_web_server: update expected assertion for static storage 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-26 16:47:36 +01:00
Azat Khuzhin
66a40bda0a Revert "Prohibit DROP from static storages" 
It is actually OK to DROP at least from read-only disk, as for the
write-once it should be OK as well, since CREATE is prohibited anyway,
so there should be no leftovers

CI: https://s3.amazonaws.com/clickhouse-test-reports/59170/c82050d1925439f0ede2b32acb5b1b8df4acae5d/stress_test__asan_.html
CI: https://s3.amazonaws.com/clickhouse-test-reports/59170/c82050d1925439f0ede2b32acb5b1b8df4acae5d/integration_tests__asan__[3_4].html

This reverts commit 082e62712c5e2f4c48d1abc6141ccfd1249d26b4.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-26 16:47:35 +01:00
Azat Khuzhin
d99fb4ce5b tests: rewrite test for MergeTree engines in s3_plain using ATTACH 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-26 16:47:35 +01:00
Azat Khuzhin
aafc149c1e Prohibit CREATE on static storages (ATTACH should be used instead) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-26 16:47:35 +01:00
Azat Khuzhin
1ee723d50d Prohibit DROP from static storages 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-26 16:47:35 +01:00
Azat Khuzhin
046a7a5df5 tests: cover DROP TABLE on s3_plain disk 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-26 16:47:35 +01:00
Azat Khuzhin
d79c1f7e1c Fix MetadataStorageFromPlainObjectStorage::iterateDirectory() (add prefix) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-26 16:47:35 +01:00
Azat Khuzhin
9ce18bbf43 Implement MetadataStorageFromPlainObjectStorageTransaction::removeDirectory (for DROP TABLE) 
This is required to remove format_version.txt for MergeTree after DROP

    (lldb) bt
    * thread 192, name = 'BgSchPool', stop reason = breakpoint 1.1
        frame 0: 0x0000000016c7f452 clickhouse`__cxa_throw(thrown_object=0x00007ffd7a53c480, tinfo=0x00000000011602a0, dest=(clickhouse`DB::Exception::~Exception() at Exception.h:40)) + 18 at cxa_exception.cpp:258
        frame 1: 0x00000000109bf709 clickhouse`DB::IMetadataTransaction::throwNotImplemented() + 137 at IMetadataStorage.h:151
        frame 2: 0x00000000109bf126 clickhouse`DB::IMetadataTransaction::removeDirectory(this=<unavailable>, (null)=<unavailable>) + 6 at IMetadataStorage.h:96
        frame 3: 0x0000000010968978 clickhouse`DB::(anonymous namespace)::RemoveRecursiveObjectStorageOperation::removeMetadataRecursive(this=0x00007ffd7a7327a0, tx=std::__1::shared_ptr<DB::IMetadataTransaction>::element_type @ 0x00007ffd7a6f6018 strong=4 weak=1, path_to_remove="tmp_insert_all_2_2_0") + 376 at DiskObjectStorageTransaction.cpp:364
        frame 4: 0x000000001096893d clickhouse`DB::(anonymous namespace)::RemoveRecursiveObjectStorageOperation::removeMetadataRecursive(this=0x00007ffd7a7327a0, tx=std::__1::shared_ptr<DB::IMetadataTransaction>::element_type @ 0x00007ffd7a6f6018 strong=4 weak=1, path_to_remove="store/27f/27f6f9d2-ab2e-4817-b771-0e706e485b15/") + 317 at DiskObjectStorageTransaction.cpp:362
        frame 5: 0x0000000010965a7f clickhouse`DB::(anonymous namespace)::RemoveRecursiveObjectStorageOperation::execute(this=0x00007ffd7a7327a0, tx=std::__1::shared_ptr<DB::IMetadataTransaction>::element_type @ 0x00007ffd7a6f6018 strong=4 weak=1) + 95 at DiskObjectStorageTransaction.cpp:372
        frame 6: 0x0000000010963a2f clickhouse`DB::DiskObjectStorageTransaction::commit(this=0x00007ffea22b6168) + 79 at DiskObjectStorageTransaction.cpp:903
        frame 7: 0x00000000109577e5 clickhouse`DB::DiskObjectStorage::removeSharedRecursive(this=<unavailable>, path="store/27f/27f6f9d2-ab2e-4817-b771-0e706e485b15/", keep_all_batch_data=false, file_names_remove_metadata_only=size=0) + 133 at DiskObjectStorage.cpp:426
        frame 8: 0x000000001095af79 clickhouse`DB::DiskObjectStorage::removeRecursive(this=<unavailable>, path=<unavailable>) + 57 at DiskObjectStorage.h:79
      * frame 9: 0x0000000010b5c3d5 clickhouse`DB::DatabaseCatalog::dropTableFinally(this=0x00007ffff7096800, table=0x00007ffe853e1bd0) + 1365 at DatabaseCatalog.cpp:1317
        frame 10: 0x0000000010b5b7e7 clickhouse`DB::DatabaseCatalog::dropTableDataTask(this=0x00007ffff7096800) + 2375 at DatabaseCatalog.cpp:1270

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-26 16:47:35 +01:00
Azat Khuzhin
f3b4f28f0d Disable mutable operations into {Replicated,}MergeTree with static storage 
Mutable operations includes:
- INSERT
- OPTIMIZE
- parts manipulation
- ...

But does not include DROP, since you need an ability to DROP it.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-26 16:47:35 +01:00
Azat Khuzhin
41ebaefeda Add a note for MetadataStorageFromPlainObjectStorage to support transactions 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-26 16:47:35 +01:00
Azat Khuzhin
6751e01cc3 Do not drop parts for static (ro/write-once) storages for Replicated engine 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-26 16:47:35 +01:00
Azat Khuzhin
e56728ae6c tests: add s3_plain disk 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-01-26 16:47:35 +01:00
Dmitry Novik
6c5057c4f2
Merge pull request #58919 from ClickHouse/analyzer-inj-func-elemination 
Analyzer: Support GROUP BY injective function elimination
 2024-01-26 16:45:09 +01:00
kssenii
5cd757dd9a Merge remote-tracking branch 'origin/master' into s3-queue-parallelize-ordered-mode 2024-01-26 16:33:22 +01:00
Kseniia Sumarokova
99653fc778
Merge branch 'master' into broken-projections-better-handling 2024-01-26 16:27:23 +01:00
kssenii
ee15b6e646 Fix 2024-01-26 16:26:59 +01:00
Dmitry Novik
a9d2e13697 Handle CTEs more careafully 2024-01-26 15:20:09 +00:00
Dmitry Novik
e38face46b Fix style 2024-01-26 15:20:09 +00:00
Dmitry Novik
778f4371c2 Cleanup the code 2024-01-26 15:20:09 +00:00
Dmitry Novik
e3c07d4562 Analyzer: Fix CTE name clash resolution 2024-01-26 15:20:09 +00:00