thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-10 09:32:06 +00:00
Code Issues Packages Projects Releases Wiki Activity
149,497 Commits 9,098 Branches 1,774 Tags 2 GiB
f08cb90fe3
Commit Graph

149497 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Alexey Milovidov
fbd3e122e1
Merge pull request #67512 from ClickHouse/divanik/fix_retry_integration_tests_decorator 
[Green CI] Test `test_storage_azure_blob_storage/test.py` is flaky
 2024-08-04 15:46:55 +00:00
Alexey Milovidov
623665dc45
Merge pull request #67739 from azat/system.clusters-mask 
Fill only selected columns from system.clusters
 2024-08-04 15:46:01 +00:00
Alexey Milovidov
592f966c2e
Merge pull request #67740 from azat/bump-nuraft 
Bump NuRaft (to properly catch thread exceptions)
 2024-08-04 15:45:41 +00:00
Alexey Milovidov
7073022f11
Merge pull request #67623 from ClickHouse/fix-logical-error-in-cached-about-expected-read-size 
Fix `Logical error: 'file_offset_of_buffer_end <= read_until_position'` in fs cache
 2024-08-04 15:44:46 +00:00
Azat Khuzhin
eed2edd7db Comment certificateFile/privateKeyFile/dhParamsFile in keeper config 
Otherwise you will get annoying messages at startup:

    2024.07.02 10:03:38.331593 [ 1 ] {} <Error> CertificateReloader: Cannot obtain modification time for certificate file /etc/clickhouse-keeper/server.crt, skipping update. errno: 2, strerror: 0
    2024.07.02 10:03:38.331658 [ 1 ] {} <Error> CertificateReloader: Cannot obtain modification time for key file /etc/clickhouse-keeper/server.key, skipping update. errno: 2, strerror: 0
    2024.07.02 10:03:38.341085 [ 1 ] {} <Error> CertificateReloader: Poco::Exception. Code: 1000, e.code() = 0, SSL context exception: Error loading private key from file /etc/clickhouse-keeper/server.key: error:80000002:system library::No such file or directory

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-08-04 16:20:42 +02:00
Azat Khuzhin
1f2bb6676f Remove outdated comments about workaround for #64086 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-08-04 16:14:05 +02:00
Azat Khuzhin
dea95e6c33 Ensure that llvm-symbolizer is used for symbolizing sanitizer reports 
Since you don't want to fallback to addr2line:

    # addr2line
    $ time /bin/test
    set_flag_impl: Success
    set_flag_if: Success
    /usr/bin/addr2line: DWARF error: invalid or unhandled FORM value: 0x23
    ==================
    WARNING: ThreadSanitizer: data race (pid=18)
    ...
    real    3m8.580s
    user    0m21.967s
    sys     0m40.628s

    # llvm-symbolizer
    $ time ./test
    set_flag_impl: Success
    set_flag_if: Success
    ==================
    WARNING: ThreadSanitizer: data race (pid=24884)
    real    0m0.028s
    user    0m0.003s
    sys     0m0.006s

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-08-04 16:14:05 +02:00
Azat Khuzhin
9f31488e50 Fix dictionary hang in case of CANNOT_SCHEDULE_TASK while loading 
On CI you can find that 01747_executable_pool_dictionary_implicit_key
can hang [1], it is possible due to after CANNOT_SCHEDULE_TASK the async
loading will hang:

    2024.07.18 03:56:32.365226 [ 6138 ] {6206a18f-668c-4a5c-a5ad-07f577220762} <Trace> ExternalDictionariesLoader: Will load the object 'executable_pool_simple_implicit_key' in background, force = false, loading_id = 2
    2024.07.18 03:56:32.368005 [ 6138 ] {6206a18f-668c-4a5c-a5ad-07f577220762} <Error> executeQuery: Code: 439. DB::Exception: Cannot schedule a task: fault injected (threads=766, jobs=746): In scope SELECT dictGet('executable_pool_simple_implicit_key', 'a', toUInt64(1)). (CANNOT_SCHEDULE_TASK) (version 24.7.1.2241) (from [::1]:56446) (comment: 01747_executable_pool_dictionary_implicit_key.sql) (in query: SELECT dictGet('executable_pool_simple_implicit_key', 'a', toUInt64(1));), Stack trace (when copying this message, always include the lines below):
    0. /build/contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x0000000015f8a292
    1. /build/src/Common/Exception.cpp:110: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000c3df6b9
    2. /build/contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000006de714c
    3. /build/contrib/llvm-project/libcxx/include/vector:438: DB::Exception::Exception<String const&, unsigned long, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<unsigned long>::type, std::type_identity<unsigned long&>::type>, String const&, unsigned long&&, unsigned long&) @ 0x000000000c4838eb
    4. /build/src/Common/ThreadPool.cpp:0: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda'(String const&)::operator()(String const&) const @ 0x000000000c4832d3
    5. /build/src/Common/ThreadPool.cpp:186: void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool) @ 0x000000000c47e7db
    6. /build/contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x000000000c47ec8d
    7. /build/contrib/llvm-project/libcxx/include/__functional/function.h:818: ? @ 0x000000001114b16e
    8. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:701: DB::ExternalLoader::LoadingDispatcher::startLoading(DB::ExternalLoader::LoadingDispatcher::Info&, bool, unsigned long) @ 0x0000000011147733
    9. /build/src/Interpreters/ExternalLoader.cpp:837: DB::ExternalLoader::LoadingDispatcher::loadImpl(String const&, std::chrono::duration<long long, std::ratio<1l, 1000l>>, bool, std::unique_lock<std::mutex>&)::'lambda'()::operator()() const @ 0x0000000011158bf9
    10. /build/contrib/llvm-project/libcxx/include/__mutex_base:397: DB::ExternalLoader::LoadingDispatcher::loadImpl(String const&, std::chrono::duration<long long, std::ratio<1l, 1000l>>, bool, std::unique_lock<std::mutex>&) @ 0x00000000111588bc
    11. /build/src/Interpreters/ExternalLoader.cpp:604: DB::ExternalLoader::LoadResult DB::ExternalLoader::LoadingDispatcher::tryLoad<DB::ExternalLoader::LoadResult>(String const&, std::chrono::duration<long long, std::ratio<1l, 1000l>>) @ 0x00000000111440bf
    12. /build/src/Interpreters/ExternalLoader.cpp:1381: std::shared_ptr<DB::IExternalLoadable const> DB::ExternalLoader::load<std::shared_ptr<DB::IExternalLoadable const>, void>(String const&) const @ 0x00000000111442f5
    13. /build/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:587: DB::ExternalDictionariesLoader::getDictionary(String const&, std::shared_ptr<DB::Context const>) const @ 0x0000000011141028
    14. /build/src/Functions/FunctionsExternalDictionaries.h:76: DB::FunctionDictHelper::getDictionary(String const&) @ 0x00000000071d28ec
    ...
    2024.07.18 03:58:29.000900 [ 48468 ] {8cf63d7e-dcbf-4af6-bd7c-0e1789ddce3b} <Debug> executeQuery: (from [::1]:40410) (comment: 01747_executable_pool_dictionary_implicit_key.sql) SELECT dictGet('executable_pool_simple_implicit_key', 'a', toUInt64(1)); (stage: Complete)
    # and no more rows for 8cf63d7e-dcbf-4af6-bd7c-0e1789ddce3b

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/66495/bc029ed8207ac75e96e9cb48cb79d27a9ffa4e2f/stress_test__debug_.html

The problem that it should be properly cancelled, otherwise it will not
be loaded in loadImpl(), but will be waited.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-08-04 15:48:59 +02:00
Alexey Milovidov
c629e2af50
Merge pull request #67741 from ClickHouse/arrayWithConstant-safety-thresholds 
Better safety thresholds in `arrayWithConstant` function
 2024-08-04 13:22:06 +00:00
Alexey Milovidov
06f83022da
Merge pull request #67324 from ClickHouse/fix-json-recursion 
Fix stack overflow in too deep JSON
 2024-08-04 13:21:44 +00:00
Kseniia Sumarokova
881ac7208e
Merge pull request #67361 from ClickHouse/add-no-distributed-cache-tag 
Add no-distributed-cache tag in tests
 2024-08-04 12:36:31 +00:00
Azat Khuzhin
062490e1b4 tests: fix test_ttl_move::test_alter_with_merge_work flakiness 
Increase timeout for TTL DELETE, since otherwise if other routines will
take too long, the part will be removed when it should be still be on
"external" disk:

    2024.08.04 03:48:53.803032 [ 622 ] {} <Information> default.mt_test_alter_with_merge_work_1722743323 (9dc6904a-f082-4f06-be7a-efe4733e811c): Will drop empty part all_1_3_4_4

And this is how part_log looks like:

    SELECT
        event_time,
        event_type,
        rows,
        part_name,
        error,
        database,
        disk_name
    FROM system.part_log
    WHERE `table` = 'mt_test_alter_with_merge_work_1722743323'
    ORDER BY event_time ASC

    Query id: a118b3cd-e4fe-45a5-b675-d73bdd887d79

        ┌──────────event_time─┬─event_type─┬─rows─┬─part_name───┬─error─┬─database─┬─disk_name─┐
     1. │ 2024-08-04 03:48:44 │ NewPart    │    2 │ all_1_1_0   │     0 │ default  │ jbod1     │
     2. │ 2024-08-04 03:48:44 │ NewPart    │    2 │ all_2_2_0   │     0 │ default  │ jbod2     │
     3. │ 2024-08-04 03:48:45 │ NewPart    │    2 │ all_3_3_0   │     0 │ default  │ jbod1     │
     4. │ 2024-08-04 03:48:46 │ MutatePart │    2 │ all_1_1_0_4 │     0 │ default  │ jbod1     │
     5. │ 2024-08-04 03:48:46 │ MutatePart │    2 │ all_2_2_0_4 │     0 │ default  │ jbod2     │
     6. │ 2024-08-04 03:48:46 │ MutatePart │    2 │ all_3_3_0_4 │     0 │ default  │ jbod1     │
     7. │ 2024-08-04 03:48:47 │ MovePart   │    2 │ all_1_1_0_4 │     0 │ default  │ external  │
     8. │ 2024-08-04 03:48:47 │ MovePart   │    2 │ all_3_3_0_4 │     0 │ default  │ jbod2     │
     9. │ 2024-08-04 03:48:47 │ MergeParts │    6 │ all_1_3_1_4 │     0 │ default  │ jbod2     │
    10. │ 2024-08-04 03:48:48 │ MovePart   │    6 │ all_1_3_1_4 │     0 │ default  │ external  │
    11. │ 2024-08-04 03:48:52 │ MergeParts │    4 │ all_1_3_2_4 │     0 │ default  │ external  │
    12. │ 2024-08-04 03:48:53 │ MergeParts │    0 │ all_1_3_3_4 │     0 │ default  │ external  │ # rows==0
    13. │ 2024-08-04 03:48:53 │ MergeParts │    0 │ all_1_3_4_4 │     0 │ default  │ external  │
        └─────────────────────┴────────────┴──────┴─────────────┴───────┴──────────┴───────────┘

CI: https://s3.amazonaws.com/clickhouse-test-reports/66671/2f00c962711e13ca00af324366421fe4593b4ce6/integration_tests__tsan__[5_6].html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-08-04 14:25:18 +02:00
Azat Khuzhin
47dbc5e05b tests: add debug info into test_ttl_move::test_alter_with_merge_work 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-08-04 14:16:40 +02:00
Azat Khuzhin
f0aaac3bd1 tests: remove useless retries from test_ttl_move::test_alter_with_merge_work 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-08-04 14:14:42 +02:00
Robert Schulze
e4c165e919
Merge pull request #67103 from bigo-sg/fix_map_from_arrays 
Function `mapFromArrays`: Allow `Map(K, V)` and `Array(Nullable(T))` as 1st arguments
 2024-08-04 09:54:50 +00:00
Robert Schulze
80b41a3edd
Merge remote-tracking branch 'ClickHouse/master' into input_row_count_remainders 2024-08-04 09:53:00 +00:00
Robert Schulze
27db715761
Incorporate review feedback 2024-08-04 09:52:36 +00:00
Robert Schulze
ee8079b039
Merge remote-tracking branch 'ClickHouse/master' into icu-s390x 2024-08-04 09:38:36 +00:00
Kseniia Sumarokova
4f1c2ad277
Merge pull request #67459 from ClickHouse/write-buffer-from-s3-fix-dtor 
Fix wait for tasks in ~WriteBufferFromS3 in case WriteBuffer was cancelled
 2024-08-04 09:27:24 +00:00
Kseniia Sumarokova
15e1217beb
Merge branch 'master' into fix-busy-loop 2024-08-04 11:26:39 +02:00
Azat Khuzhin
dc527b6fd1 tests: detailed errors for RabbitMQ startup 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-08-04 09:07:05 +02:00
Azat Khuzhin
264be9c598 tests: increase timeout for RabbitMQ startup 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-08-04 09:06:50 +02:00
Azat Khuzhin
59e5a09b49 Use RabbitMQ without management in tests (attempt to improve startup) 
Sometimes startup can take ~90 seconds [1]:

    2024-08-03 23:11:38.756067+00:00 [info] <0.9.0> Time to start RabbitMQ: 94651980 us

Unlike normally ~10 [2]:

    2024-08-04 00:33:07.016137+00:00 [info] <0.9.0> Time to start RabbitMQ: 10082489 us

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/67737/b4e3bbcb82158bea4f5db1d9f5c28cfb741d1d51/integration_tests__asan__old_analyzer__[4_6].html
  [2]: https://s3.amazonaws.com/clickhouse-test-reports/66671/2f00c962711e13ca00af324366421fe4593b4ce6/integration_tests__asan__old_analyzer__[4_6].html

I've tried locally, the difference is very small, 3135665 us
(+management) vs (2740747 us), but still something, and who knows how it
works under pressure.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-08-04 09:04:41 +02:00
Nikita Fomichev
c6c4b313e6
Merge pull request #67738 from azat/tests/fix-retries 
Fix test retries
 2024-08-04 06:33:48 +00:00
Peter Nguyen
a31e2a4a5a
Merge branch 'ClickHouse:master' into aliasFeature 2024-08-03 21:38:42 -06:00
Alexey Milovidov
3efa208680
Merge pull request #67729 from ClickHouse/fix-typo 
Fix a typo
 2024-08-03 23:04:41 +00:00
Alexey Milovidov
6a20f777f6
Merge pull request #67428 from ClickHouse/enable-text-log 
Enable `text_log` by default
 2024-08-03 23:02:40 +00:00
Alexey Milovidov
e7686494f5
Merge pull request #67727 from ClickHouse/fix-host-resolver-ubsan 
Fix strange code in HostResolvePool
 2024-08-03 22:41:14 +00:00
Alexey Milovidov
185b6a54da Merge with master 2024-08-04 00:34:58 +02:00
Alexey Milovidov
d37a4567f6 Merge branch 'master' into fix-json-recursion 2024-08-04 00:30:29 +02:00
Alexey Milovidov
3bea8b4e25 Merge branch 'master' into add-assert-into-timer-descriptor 2024-08-04 00:29:22 +02:00
Alexey Milovidov
b8050cc471
Merge pull request #67702 from ClickHouse/fix-timer-descriptor-locks 
Fix locking inside TimerDescriptor
 2024-08-04 00:28:23 +02:00
Alexey Milovidov
a051eb2a5d Fix tests 2024-08-04 00:24:17 +02:00
Alexey Milovidov
6b300ab44f Merge branch 'master' into fix-log-and-exception-messages-formatting 2024-08-04 00:23:18 +02:00
Alexey Milovidov
4e3202d906
Merge pull request #67645 from ClickHouse/fix-race-condition-in-settings-processlist 
Fix race condition between ProcessList and Settings
 2024-08-03 22:21:49 +00:00
Alexey Milovidov
aa3b4595fa
Merge pull request #67701 from ClickHouse/improve-dashboard 
Improve the dashboard
 2024-08-03 22:16:07 +00:00
Azat Khuzhin
2f00c96271 Support true UUID type for MongoDB engine 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-08-03 23:51:41 +02:00
Azat Khuzhin
6013e4b81d poco/MongoDB: Support broader UUID types (generated by python uuid.UUID) 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-08-03 23:51:41 +02:00
Alexey Milovidov
7a066a6505 Add a test 2024-08-03 23:48:51 +02:00
Alexey Milovidov
8562a6106c Better safety thresholds in arrayWithConstant 2024-08-03 23:45:18 +02:00
Azat Khuzhin
9d0e066cda Bump NuRaft (to properly catch thread exceptions) 
Refs: https://github.com/ClickHouse/NuRaft/pull/75
Refs: https://github.com/eBay/NuRaft/pull/525
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-08-03 23:41:50 +02:00
Azat Khuzhin
f06ae2f551 Fill only selected columns from system.clusters 
Some of them pretty heavy, i.e. is_active for ReplicatedDatabase

This should fix 02903_rmt_retriable_merge_exception flakiness [1].

 [1]: https://s3.amazonaws.com/clickhouse-test-reports/67687/89c47df559ba23d988f8af3c342e0c8d5531f4b8/fast_test.html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-08-03 23:37:26 +02:00
Azat Khuzhin
a197502341 Fix test retries 
Should fix issues like:
- 02494_zero_copy_projection_cancel_fetch - https://s3.amazonaws.com/clickhouse-test-reports/67719/40cd5467c18d65a6624d273ac1a8fd9cc9257d8c/stateless_tests__tsan__s3_storage__[4_4].html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2024-08-03 23:11:20 +02:00
Alexey Milovidov
d9d30d1004
Merge pull request #67295 from canhld94/fix_empty_tuple_in_lhs_in 
Fix hung query when using empty tuple as lhs of function IN
 2024-08-03 20:57:59 +00:00
Max K
3dedd8d76b CI: Minor refactoring in ci_utils 2024-08-03 21:19:51 +02:00
Alexey Milovidov
02b8d563e3
Merge pull request #67720 from vitlibar/fix-completing-restore-on-cluster 
Fix completion of RESTORE ON CLUSTER
 2024-08-03 18:58:32 +00:00
Alexey Milovidov
c91504ffb4
Merge pull request #67719 from azat/tests/01042_system_reload_dictionary_reloads_completely 
Fix 01042_system_reload_dictionary_reloads_completely flakiness
 2024-08-03 18:49:21 +00:00
Alexey Milovidov
54fa1f14bb Merge branch 'master' into improve-dashboard 2024-08-03 20:37:41 +02:00
Alexey Milovidov
2305cae149
Merge pull request #67564 from ClickHouse/miscellaneous-2 
Miscellaneous
 2024-08-03 18:16:35 +00:00
Alexey Milovidov
5e0d13e38b Merge branch 'master' into fix-log-and-exception-messages-formatting 2024-08-03 19:25:33 +02:00