thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-12 09:22:05 +00:00
Code Issues Packages Projects Releases Wiki Activity
149,633 Commits 9,297 Branches 1,782 Tags 2.1 GiB
37a6bd25f6
Commit Graph

149633 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Alexey Milovidov
fbd3e122e1
Merge pull request #67512 from ClickHouse/divanik/fix_retry_integration_tests_decorator 
[Green CI] Test `test_storage_azure_blob_storage/test.py` is flaky
 2024-08-04 15:46:55 +00:00
Alexey Milovidov
623665dc45
Merge pull request #67739 from azat/system.clusters-mask 
Fill only selected columns from system.clusters
 2024-08-04 15:46:01 +00:00
Alexey Milovidov
592f966c2e
Merge pull request #67740 from azat/bump-nuraft 
Bump NuRaft (to properly catch thread exceptions)
 2024-08-04 15:45:41 +00:00
Alexey Milovidov
7073022f11
Merge pull request #67623 from ClickHouse/fix-logical-error-in-cached-about-expected-read-size 
Fix `Logical error: 'file_offset_of_buffer_end <= read_until_position'` in fs cache
 2024-08-04 15:44:46 +00:00
Azat Khuzhin
eed2edd7db Comment certificateFile/privateKeyFile/dhParamsFile in keeper config 
Otherwise you will get annoying messages at startup:

    2024.07.02 10:03:38.331593 [ 1 ] {} <Error> CertificateReloader: Cannot obtain modification time for certificate file /etc/clickhouse-keeper/server.crt, skipping update. errno: 2, strerror: 0
    2024.07.02 10:03:38.331658 [ 1 ] {} <Error> CertificateReloader: Cannot obtain modification time for key file /etc/clickhouse-keeper/server.key, skipping update. errno: 2, strerror: 0
    2024.07.02 10:03:38.341085 [ 1 ] {} <Error> CertificateReloader: Poco::Exception. Code: 1000, e.code() = 0, SSL context exception: Error loading private key from file /etc/clickhouse-keeper/server.key: error:80000002:system library::No such file or directory

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-08-04 16:20:42 +02:00
Azat Khuzhin
1f2bb6676f Remove outdated comments about workaround for #64086 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-08-04 16:14:05 +02:00
Azat Khuzhin
dea95e6c33 Ensure that llvm-symbolizer is used for symbolizing sanitizer reports 
Since you don't want to fallback to addr2line:

    # addr2line
    $ time /bin/test
    set_flag_impl: Success
    set_flag_if: Success
    /usr/bin/addr2line: DWARF error: invalid or unhandled FORM value: 0x23
    ==================
    WARNING: ThreadSanitizer: data race (pid=18)
    ...
    real    3m8.580s
    user    0m21.967s
    sys     0m40.628s

    # llvm-symbolizer
    $ time ./test
    set_flag_impl: Success
    set_flag_if: Success
    ==================
    WARNING: ThreadSanitizer: data race (pid=24884)
    real    0m0.028s
    user    0m0.003s
    sys     0m0.006s

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-08-04 16:14:05 +02:00
Azat Khuzhin
9f31488e50 Fix dictionary hang in case of CANNOT_SCHEDULE_TASK while loading 
On CI you can find that 01747_executable_pool_dictionary_implicit_key
can hang [1], it is possible due to after CANNOT_SCHEDULE_TASK the async
loading will hang:

    2024.07.18 03:56:32.365226 [ 6138 ] {6206a18f-668c-4a5c-a5ad-07f577220762} <Trace> ExternalDictionariesLoader: Will load the object 'executable_pool_simple_implicit_key' in background, force = false, loading_id = 2
    2024.07.18 03:56:32.368005 [ 6138 ] {6206a18f-668c-4a5c-a5ad-07f577220762} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=766, jobs=746): In scope SELECT dictGet('executable_pool_simple_implicit_key', 'a', toUInt64(1)). (CANNOT_SCHEDULE_TASK) (version 24.7.1.2241) (from [::1]:56446) (comment: 01747_executable_pool_dictionary_implicit_key.sql) (in query: SELECT dictGet('executable_pool_simple_implicit_key', 'a', toUInt64(1));), Stack trace (when copying this message, always include the lines below):
    0. /build/contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x0000000015f8a292
    1. /build/src/Common/Exception.cpp:110: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c3df6b9
    2. /build/contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000006de714c
    3. /build/contrib/llvm-project/libcxx/include/vector:438: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x000000000c4838eb
    4. /build/src/Common/ThreadPool.cpp:0: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x000000000c4832d3
    5. /build/src/Common/ThreadPool.cpp:186: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x000000000c47e7db
    6. /build/contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x000000000c47ec8d
    7. /build/contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x000000001114b16e
    8. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:701: DB::ExternalLoader::LoadingDispatcher::startLoading(DB::ExternalLoader::LoadingDispatcher::Info&, bool, unsigned long) @ 0x0000000011147733
    9. /build/src/Interpreters/ExternalLoader.cpp:837: DB::ExternalLoader::LoadingDispatcher::loadImpl(String const&, std::chrono::duration<long long, std::ratio<1l, 1000l>>, bool, std::unique_lock<std::mutex>&)::'lambda'()::operator()() const @ 0x0000000011158bf9
    10. /build/contrib/llvm-project/libcxx/include/__mutex_base:397: DB::ExternalLoader::LoadingDispatcher::loadImpl(String const&, std::chrono::duration<long long, std::ratio<1l, 1000l>>, bool, std::unique_lock<std::mutex>&) @ 0x00000000111588bc
    11. /build/src/Interpreters/ExternalLoader.cpp:604: DB::ExternalLoader::LoadResult DB::ExternalLoader::LoadingDispatcher::tryLoad<DB::ExternalLoader::LoadResult>(String const&, std::chrono::duration<long long, std::ratio<1l, 1000l>>) @ 0x00000000111440bf
    12. /build/src/Interpreters/ExternalLoader.cpp:1381: std::shared_ptr<DB::IExternalLoadable const> DB::ExternalLoader::load<std::shared_ptr<DB::IExternalLoadable const>, void>(String const&) const @ 0x00000000111442f5
    13. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:587: DB::ExternalDictionariesLoader::getDictionary(String const&, std::shared_ptr<DB::Context const>) const @ 0x0000000011141028
    14. /build/src/Functions/FunctionsExternalDictionaries.h:76: DB::FunctionDictHelper::getDictionary(String const&) @ 0x00000000071d28ec
    ...
    2024.07.18 03:58:29.000900 [ 48468 ] {8cf63d7e-dcbf-4af6-bd7c-0e1789ddce3b} <Debug> executeQuery: (from [::1]:40410) (comment: 01747_executable_pool_dictionary_implicit_key.sql) SELECT dictGet('executable_pool_simple_implicit_key', 'a', toUInt64(1)); (stage: Complete)
    # and no more rows for 8cf63d7e-dcbf-4af6-bd7c-0e1789ddce3b

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/66495/bc029ed8207ac75e96e9cb48cb79d27a9ffa4e2f/stress_test__debug_.html

The problem that it should be properly cancelled, otherwise it will not
be loaded in loadImpl(), but will be waited.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-08-04 15:48:59 +02:00
Alexey Milovidov
c629e2af50
Merge pull request #67741 from ClickHouse/arrayWithConstant-safety-thresholds 
Better safety thresholds in `arrayWithConstant` function
 2024-08-04 13:22:06 +00:00
Alexey Milovidov
06f83022da
Merge pull request #67324 from ClickHouse/fix-json-recursion 
Fix stack overflow in too deep JSON
 2024-08-04 13:21:44 +00:00
Kseniia Sumarokova
881ac7208e
Merge pull request #67361 from ClickHouse/add-no-distributed-cache-tag 
Add no-distributed-cache tag in tests
 2024-08-04 12:36:31 +00:00
Azat Khuzhin
062490e1b4 tests: fix test_ttl_move::test_alter_with_merge_work flakiness 
Increase timeout for TTL DELETE, since otherwise if other routines will
take too long, the part will be removed when it should be still be on
"external" disk:

    2024.08.04 03:48:53.803032 [ 622 ] {} <Information> default.mt_test_alter_with_merge_work_1722743323 (9dc6904a-f082-4f06-be7a-efe4733e811c): Will drop empty part all_1_3_4_4

And this is how part_log looks like:

    SELECT
        event_time,
        event_type,
        rows,
        part_name,
        error,
        database,
        disk_name
    FROM system.part_log
    WHERE `table` = 'mt_test_alter_with_merge_work_1722743323'
    ORDER BY event_time ASC

    Query id: a118b3cd-e4fe-45a5-b675-d73bdd887d79

        ┌──────────event_time─┬─event_type─┬─rows─┬─part_name───┬─error─┬─database─┬─disk_name─┐
     1. │ 2024-08-04 03:48:44 │ NewPart    │    2 │ all_1_1_0   │     0 │ default  │ jbod1     │
     2. │ 2024-08-04 03:48:44 │ NewPart    │    2 │ all_2_2_0   │     0 │ default  │ jbod2     │
     3. │ 2024-08-04 03:48:45 │ NewPart    │    2 │ all_3_3_0   │     0 │ default  │ jbod1     │
     4. │ 2024-08-04 03:48:46 │ MutatePart │    2 │ all_1_1_0_4 │     0 │ default  │ jbod1     │
     5. │ 2024-08-04 03:48:46 │ MutatePart │    2 │ all_2_2_0_4 │     0 │ default  │ jbod2     │
     6. │ 2024-08-04 03:48:46 │ MutatePart │    2 │ all_3_3_0_4 │     0 │ default  │ jbod1     │
     7. │ 2024-08-04 03:48:47 │ MovePart   │    2 │ all_1_1_0_4 │     0 │ default  │ external  │
     8. │ 2024-08-04 03:48:47 │ MovePart   │    2 │ all_3_3_0_4 │     0 │ default  │ jbod2     │
     9. │ 2024-08-04 03:48:47 │ MergeParts │    6 │ all_1_3_1_4 │     0 │ default  │ jbod2     │
    10. │ 2024-08-04 03:48:48 │ MovePart   │    6 │ all_1_3_1_4 │     0 │ default  │ external  │
    11. │ 2024-08-04 03:48:52 │ MergeParts │    4 │ all_1_3_2_4 │     0 │ default  │ external  │
    12. │ 2024-08-04 03:48:53 │ MergeParts │    0 │ all_1_3_3_4 │     0 │ default  │ external  │ # rows==0
    13. │ 2024-08-04 03:48:53 │ MergeParts │    0 │ all_1_3_4_4 │     0 │ default  │ external  │
        └─────────────────────┴────────────┴──────┴─────────────┴───────┴──────────┴───────────┘

CI: https://s3.amazonaws.com/clickhouse-test-reports/66671/2f00c962711e13ca00af324366421fe4593b4ce6/integration_tests__tsan__[5_6].html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-08-04 14:25:18 +02:00
Azat Khuzhin
47dbc5e05b tests: add debug info into test_ttl_move::test_alter_with_merge_work 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-08-04 14:16:40 +02:00
Azat Khuzhin
f0aaac3bd1 tests: remove useless retries from test_ttl_move::test_alter_with_merge_work 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-08-04 14:14:42 +02:00
Robert Schulze
e4c165e919
Merge pull request #67103 from bigo-sg/fix_map_from_arrays 
Function `mapFromArrays`: Allow `Map(K, V)` and `Array(Nullable(T))` as 1st arguments
 2024-08-04 09:54:50 +00:00
Robert Schulze
80b41a3edd
Merge remote-tracking branch 'ClickHouse/master' into input_row_count_remainders 2024-08-04 09:53:00 +00:00
Robert Schulze
27db715761
Incorporate review feedback 2024-08-04 09:52:36 +00:00
Robert Schulze
ee8079b039
Merge remote-tracking branch 'ClickHouse/master' into icu-s390x 2024-08-04 09:38:36 +00:00
Kseniia Sumarokova
4f1c2ad277
Merge pull request #67459 from ClickHouse/write-buffer-from-s3-fix-dtor 
Fix wait for tasks in ~WriteBufferFromS3 in case WriteBuffer was cancelled
 2024-08-04 09:27:24 +00:00
Kseniia Sumarokova
15e1217beb
Merge branch 'master' into fix-busy-loop 2024-08-04 11:26:39 +02:00
Azat Khuzhin
dc527b6fd1 tests: detailed errors for RabbitMQ startup 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-08-04 09:07:05 +02:00
Azat Khuzhin
264be9c598 tests: increase timeout for RabbitMQ startup 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-08-04 09:06:50 +02:00
Azat Khuzhin
59e5a09b49 Use RabbitMQ without management in tests (attempt to improve startup) 
Sometimes startup can take ~90 seconds [1]:

    2024-08-03 23:11:38.756067+00:00 [info] <0.9.0> Time to start RabbitMQ: 94651980 us

Unlike normally ~10 [2]:

    2024-08-04 00:33:07.016137+00:00 [info] <0.9.0> Time to start RabbitMQ: 10082489 us

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/67737/b4e3bbcb82158bea4f5db1d9f5c28cfb741d1d51/integration_tests__asan__old_analyzer__[4_6].html
  [2]: https://s3.amazonaws.com/clickhouse-test-reports/66671/2f00c962711e13ca00af324366421fe4593b4ce6/integration_tests__asan__old_analyzer__[4_6].html

I've tried locally, the difference is very small, 3135665 us
(+management) vs (2740747 us), but still something, and who knows how it
works under pressure.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-08-04 09:04:41 +02:00
Nikita Fomichev
c6c4b313e6
Merge pull request #67738 from azat/tests/fix-retries 
Fix test retries
 2024-08-04 06:33:48 +00:00
Peter Nguyen
a31e2a4a5a
Merge branch 'ClickHouse:master' into aliasFeature 2024-08-03 21:38:42 -06:00
Alexey Milovidov
3efa208680
Merge pull request #67729 from ClickHouse/fix-typo 
Fix a typo
 2024-08-03 23:04:41 +00:00
Alexey Milovidov
6a20f777f6
Merge pull request #67428 from ClickHouse/enable-text-log 
Enable `text_log` by default
 2024-08-03 23:02:40 +00:00
Alexey Milovidov
e7686494f5
Merge pull request #67727 from ClickHouse/fix-host-resolver-ubsan 
Fix strange code in HostResolvePool
 2024-08-03 22:41:14 +00:00
Alexey Milovidov
185b6a54da Merge with master 2024-08-04 00:34:58 +02:00
Alexey Milovidov
d37a4567f6 Merge branch 'master' into fix-json-recursion 2024-08-04 00:30:29 +02:00
Alexey Milovidov
3bea8b4e25 Merge branch 'master' into add-assert-into-timer-descriptor 2024-08-04 00:29:22 +02:00
Alexey Milovidov
b8050cc471
Merge pull request #67702 from ClickHouse/fix-timer-descriptor-locks 
Fix locking inside TimerDescriptor
 2024-08-04 00:28:23 +02:00
Alexey Milovidov
a051eb2a5d Fix tests 2024-08-04 00:24:17 +02:00
Alexey Milovidov
6b300ab44f Merge branch 'master' into fix-log-and-exception-messages-formatting 2024-08-04 00:23:18 +02:00
Alexey Milovidov
4e3202d906
Merge pull request #67645 from ClickHouse/fix-race-condition-in-settings-processlist 
Fix race condition between ProcessList and Settings
 2024-08-03 22:21:49 +00:00
Alexey Milovidov
aa3b4595fa
Merge pull request #67701 from ClickHouse/improve-dashboard 
Improve the dashboard
 2024-08-03 22:16:07 +00:00
Azat Khuzhin
2f00c96271 Support true UUID type for MongoDB engine 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-08-03 23:51:41 +02:00
Azat Khuzhin
6013e4b81d poco/MongoDB: Support broader UUID types (generated by python uuid.UUID) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-08-03 23:51:41 +02:00
Alexey Milovidov
7a066a6505 Add a test 2024-08-03 23:48:51 +02:00
Alexey Milovidov
8562a6106c Better safety thresholds in arrayWithConstant 2024-08-03 23:45:18 +02:00
Azat Khuzhin
9d0e066cda Bump NuRaft (to properly catch thread exceptions) 
Refs: https://github.com/ClickHouse/NuRaft/pull/75
Refs: https://github.com/eBay/NuRaft/pull/525
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-08-03 23:41:50 +02:00
Azat Khuzhin
f06ae2f551 Fill only selected columns from system.clusters 
Some of them pretty heavy, i.e. is_active for ReplicatedDatabase

This should fix 02903_rmt_retriable_merge_exception flakiness [1].

 [1]: https://s3.amazonaws.com/clickhouse-test-reports/67687/89c47df559ba23d988f8af3c342e0c8d5531f4b8/fast_test.html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-08-03 23:37:26 +02:00
Azat Khuzhin
a197502341 Fix test retries 
Should fix issues like:
- 02494_zero_copy_projection_cancel_fetch - https://s3.amazonaws.com/clickhouse-test-reports/67719/40cd5467c18d65a6624d273ac1a8fd9cc9257d8c/stateless_tests__tsan__s3_storage__[4_4].html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-08-03 23:11:20 +02:00
Alexey Milovidov
d9d30d1004
Merge pull request #67295 from canhld94/fix_empty_tuple_in_lhs_in 
Fix hung query when using empty tuple as lhs of function IN
 2024-08-03 20:57:59 +00:00
Max K
3dedd8d76b CI: Minor refactoring in ci_utils 2024-08-03 21:19:51 +02:00
Alexey Milovidov
02b8d563e3
Merge pull request #67720 from vitlibar/fix-completing-restore-on-cluster 
Fix completion of RESTORE ON CLUSTER
 2024-08-03 18:58:32 +00:00
Alexey Milovidov
c91504ffb4
Merge pull request #67719 from azat/tests/01042_system_reload_dictionary_reloads_completely 
Fix 01042_system_reload_dictionary_reloads_completely flakiness
 2024-08-03 18:49:21 +00:00
Alexey Milovidov
54fa1f14bb Merge branch 'master' into improve-dashboard 2024-08-03 20:37:41 +02:00
Alexey Milovidov
2305cae149
Merge pull request #67564 from ClickHouse/miscellaneous-2 
Miscellaneous
 2024-08-03 18:16:35 +00:00
Alexey Milovidov
5e0d13e38b Merge branch 'master' into fix-log-and-exception-messages-formatting 2024-08-03 19:25:33 +02:00