thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-09-24 10:40:49 +00:00
Code Issues Packages Projects Releases Wiki Activity
74,494 Commits 8,941 Branches 1,766 Tags 2 GiB
96bee8d241
Commit Graph

74494 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Azat Khuzhin
f37dac03f4 Fix lock-order-inversion between DROP TABLE for DatabaseMemory and LiveView 
CI stress founds [1], TSan report:

WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=509)
  Cycle in lock order graph: M71436201 (0x7b5800a91308) => M237700395169415472 (0x000000000000) => M71436201

  Mutex M237700395169415472 acquired here while holding mutex M71436201 in thread T723:
    3 std::__1::lock_guard<std::__1::mutex>::lock_guard(std::__1::mutex&) obj-x86_64-linux-gnu/../contrib/libcxx/include/__mutex_base:91:27 (clickhouse+0x154b0db9)
    4 DB::DatabaseWithOwnTablesBase::tryGetTable() const obj-x86_64-linux-gnu/../src/Databases/DatabasesCommon.cpp:37:21 (clickhouse+0x154b0db9)
    5 DB::DatabaseMemory::tryGetTableUUID() const obj-x86_64-linux-gnu/../src/Databases/DatabaseMemory.cpp:95:22 (clickhouse+0x15466bb5)
    6 DB::Context::resolveStorageID() const obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:2672:90 (clickhouse+0x155e6aa1)
    7 DB::JoinedTables::getLeftTableStorage() obj-x86_64-linux-gnu/../src/Interpreters/JoinedTables.cpp:200:29 (clickhouse+0x15eee962)
    8 DB::InterpreterSelectQuery::InterpreterSelectQuery() obj-x86_64-linux-gnu/../src/Interpreters/InterpreterSelectQuery.cpp:321:33 (clickhouse+0x15b792be)
    9 DB::InterpreterSelectQuery::InterpreterSelectQuery() obj-x86_64-linux-gnu/../src/Interpreters/InterpreterSelectQuery.cpp:160:7 (clickhouse+0x15b78160)
    10 DB::StorageLiveView::collectMergeableBlocks() obj-x86_64-linux-gnu/../src/Storages/LiveView/StorageLiveView.cpp:113:28 (clickhouse+0x16aae192)
    11 DB::StorageLiveView::getNewBlocks() obj-x86_64-linux-gnu/../src/Storages/LiveView/StorageLiveView.cpp:384:33 (clickhouse+0x16ab393a)
    12 DB::StorageLiveView::refresh(bool) obj-x86_64-linux-gnu/../src/Storages/LiveView/StorageLiveView.cpp:528:13 (clickhouse+0x16ab6395)
    13 DB::StorageLiveView::read() obj-x86_64-linux-gnu/../src/Storages/LiveView/StorageLiveView.cpp:545:9 (clickhouse+0x16ab6395)
    21 DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) obj-x86_64-linux-gnu/../src/Interpreters/executeQuery.cpp:950:30 (clickhouse+0x16101441)

    Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message

  Mutex M71436201 acquired here while holding mutex M237700395169415472 in thread T723:
    3 std::__1::lock_guard<std::__1::mutex>::lock_guard(std::__1::mutex&) obj-x86_64-linux-gnu/../contrib/libcxx/include/__mutex_base:91:27 (clickhouse+0x16ab5c29)
    4 DB::StorageLiveView::drop() obj-x86_64-linux-gnu/../src/Storages/LiveView/StorageLiveView.cpp:477:21 (clickhouse+0x16ab5c29)
    5 DB::DatabaseMemory::dropTable() obj-x86_64-linux-gnu/../src/Databases/DatabaseMemory.cpp:44:16 (clickhouse+0x15465cd6)
    8 DB::InterpreterDropQuery::execute() obj-x86_64-linux-gnu/../src/Interpreters/InterpreterDropQuery.cpp:62:16 (clickhouse+0x15afa679)

  [1]: https://clickhouse-test-reports.s3.yandex.net/29856/42ca2b4bb241827edf69bbd6938d6b19c31935f1/stress_test_(thread).html#fail1

v2: fix attachTableUnlocked() (as pointed by @tavplubix)
v3: use SCOPE_EXIT() and so avoid holding a lock during removing from fs
 2021-10-10 23:51:36 +03:00
Thom O'Connor
e229847862 Documentation correction for units for block size, plus some additional context for storage system recommendations 2021-10-10 14:42:06 -06:00
Alexey
b83655b348 ru translation 2021-10-10 19:38:53 +00:00
Azat Khuzhin
6c33eaee32 Fix filtering by tuple (some conditions was lost during analyzing) 
Fixes: #29281
Fixes: test_cluster_copier/test_three_nodes.py::test
 2021-10-10 22:37:52 +03:00
Alexey Milovidov
af4066c255 Remove printf 2021-10-10 21:54:15 +03:00
Nikolai Kochetov
a95c28ec4b
Merge pull request #29898 from ClickHouse/remove-native-stream 
Remove some more streams.
 2021-10-10 21:01:16 +03:00
mikael
934b72ddd1 Add FreeBSD aarch64 bits for cross-builds for CI. 2021-10-10 18:54:38 +02:00
Vitaly Baranov
1dda596689 Fix releasing query ID and session ID at the end of query processing. 2021-10-10 17:34:45 +03:00
alexey-milovidov
af5150c38b
Merge pull request #29935 from ClickHouse/remove-trash-printf 
Remove `printf` function usage.
 2021-10-10 16:11:00 +03:00
mikael
5bd2fd0ba3 Add the missing bits for FreeBSD aarch64 2021-10-10 14:52:42 +02:00
mikael
714ac8efc4 FreeBSD aarch64 doesn't implement sbrk 2021-10-10 14:52:42 +02:00
mikael
689c9cdbb9 Add uc_mcontext for FreeBSD aarch64 2021-10-10 14:52:17 +02:00
Vitaly Baranov
28459b282d Fix shutdown of AccessControlManager. Now there cannot be configuration reloading after AccessControlManager has been destroyed. 2021-10-10 13:25:24 +03:00
tavplubix
1aeb5d55dd
Grep server log even if it contains binary data (#29903) 
* grep server log even if it contains binary data

* Update cluster.py

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2021-10-10 12:55:37 +03:00
tavplubix
ca8625f1a3
Merge pull request #29923 from ClickHouse/fix_tests_replication_check 
Fix some flaky tests
 2021-10-10 12:53:54 +03:00
Azat Khuzhin
0c5d26b647 Make 01085_max_distributed_connections less flaky 2021-10-10 09:57:27 +03:00
Azat Khuzhin
38de183379 Make 00984_parser_stack_overflow less flaky 
According to query_log/trace_log the problem is real:

    2021.10.10 07:52:37.702364 [ 3736 ] {c6a219f2-3df8-4474-8324-bb307ee7a7a9} <Debug> executeQuery: (from [::1]:38096) (comment: /usr/share/clickhouse-test/queries/0_stateless/00984_parser_stack_overflow.sh) SELECT [[[...
    2021.10.10 07:53:11.407949 [ 3736 ] {c6a219f2-3df8-4474-8324-bb307ee7a7a9} <Error> executeQuery: Code: 306. DB::Exception: Maximum parse depth (1000) exceeded. Consider rising max_parser_depth ...

From trace_log:

    2021-10-10 07:52:37.958938 DB::Dwarf::findDebugInfoOffset
    2021-10-10 07:53:12.583248 DB::Dwarf::findDebugInfoOffset

CI: https://clickhouse-test-reports.s3.yandex.net/29928/2bab572caf91d2fc8962f409fea34149ecec8c95/functional_stateless_tests_(debug).html#fail1
 2021-10-10 09:57:26 +03:00
Azat Khuzhin
452b3b443d copier: add ability to configure retries and delays between them 
The following options had been introduced:
- max-table-tries
- max-shard-partition-tries
- max-shard-partition-piece-tries-for-alter
- retry-delay-ms
 2021-10-10 08:58:35 +03:00
Alexey Milovidov
103b3c91ba Remove 'printf' function usage. 2021-10-10 04:21:07 +03:00
Alexey Milovidov
2a20bf4909 Remove function bayesAB #26233 2021-10-10 04:16:32 +03:00
Alexey Milovidov
4e6ed5c45c Remove trash from SentryWriter 2021-10-10 04:10:52 +03:00
Alexey Milovidov
daf4cc6aba Whitespaces 2021-10-10 04:09:17 +03:00
Azat Khuzhin
2bab572caf Fix lock-order-inversion between periodic dictionary reload and config reload 
Integration tests found [1], TSan report:

    WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=11)
      Cycle in lock order graph: M3152 (0x7b9000000058) => M3153 (0x7b9000000438) => M3152

      Mutex M3153 acquired here while holding mutex M3152 in main thread:
        3 std::__1::lock_guard<std::__1::recursive_mutex>::lock_guard(std::__1::recursive_mutex&) obj-x86_64-linux-gnu/../contrib/libcxx/include/__mutex_base:91:27 (clickhouse+0x15716b81)
        4 DB::ExternalLoader::addConfigRepository() const obj-x86_64-linux-gnu/../src/Interpreters/ExternalLoader.cpp:1283:21 (clickhouse+0x15716b81)
        5 DB::Context::loadOrReloadDictionaries() obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:1453:62 (clickhouse+0x155d7bad)
        6 DB::Server::main()::$_1::operator()() const obj-x86_64-linux-gnu/../programs/server/Server.cpp:852:29 (clickhouse+0x9b1944c)
        13 DB::ConfigReloader::reloadIfNewer() obj-x86_64-linux-gnu/../src/Common/Config/ConfigReloader.cpp:137:13 (clickhouse+0x17045e2e)
        14 DB::ConfigReloader::ConfigReloader() obj-x86_64-linux-gnu/../src/Common/Config/ConfigReloader.cpp:33:9 (clickhouse+0x17044e51)
        16 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) obj-x86_64-linux-gnu/../programs/server/Server.cpp:803:33 (clickhouse+0x9b0c41d)
        17 Poco::Util::Application::run() obj-x86_64-linux-gnu/../contrib/poco/Util/src/Application.cpp:334:8 (clickhouse+0x19ffc08b)
        18 DB::Server::run() obj-x86_64-linux-gnu/../programs/server/Server.cpp:405:25 (clickhouse+0x9b03ebe)
        19 Poco::Util::ServerApplication::run(int, char**) obj-x86_64-linux-gnu/../contrib/poco/Util/src/ServerApplication.cpp:611:9 (clickhouse+0x1a01c246)
        20 mainEntryClickHouseServer(int, char**) obj-x86_64-linux-gnu/../programs/server/Server.cpp:183:20 (clickhouse+0x9b02576)
        21 main obj-x86_64-linux-gnu/../programs/main.cpp:372:12 (clickhouse+0x9b00a3a)

      Mutex M3152 acquired here while holding mutex M3153 in thread T2:
        3 std::__1::lock_guard<std::__1::mutex>::lock_guard() obj-x86_64-linux-gnu/../contrib/libcxx/include/__mutex_base:91:27 (clickhouse+0x155d63b8)
        4 DB::Context::getExternalDictionariesLoader() obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:1337:21 (clickhouse+0x155d63b8)
        5 DB::Context::getExternalDictionariesLoader() const obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:1332:41 (clickhouse+0x155d6359)
        6 DB::DatabaseDictionary::tryGetTable() const obj-x86_64-inux-gnu/../src/Databases/DatabaseDictionary.cpp:76:38 (clickhouse+0x157819ad)
        7 DB::DatabaseCatalog::getTableImpl() const obj-x86_64-linux-gnu/../src/Interpreters/DatabaseCatalog.cpp:285:28 (clickhouse+0x1564a1fa)
        8 DB::DatabaseCatalog::getTable() const obj-x86_64-linux-gnu/../src/Interpreters/DatabaseCatalog.cpp:656:16 (clickhouse+0x1564fa2a)
        9 DB::JoinedTables::getLeftTableStorage() obj-x86_64-linux-gnu/../src/Interpreters/JoinedTables.cpp:219:40 (clickhouse+0x15eeef45)
        10 DB::InterpreterSelectQuery::InterpreterSelectQuery() obj-x86_64-linux-gnu/../src/Interpreters/InterpreterSelectQuery.cpp:321:33 (clickhouse+0x15b792be)
        19 DB::ClickHouseDictionarySource::doInvalidateQuery() const obj-x86_64-linux-gnu/../src/Dictionaries/ClickHouseDictionarySource.cpp:207:36 (clickhouse+0x12872d2d)
        20 DB::ClickHouseDictionarySource::isModified() const obj-x86_64-linux-gnu/../src/Dictionaries/ClickHouseDictionarySource.cpp:144:25 (clickhouse+0x12872534)
        21 DB::IDictionary::isModified() const <null> (clickhouse+0x128ce39b)
        22 DB::ExternalLoader::LoadingDispatcher::reloadOutdated() obj-x86_64-linux-gnu/../src/Interpreters/ExternalLoader.cpp:660:50 (clickhouse+0x157305f7)
        23 DB::ExternalLoader::PeriodicUpdater::doPeriodicUpdates() obj-x86_64-linux-gnu/../src/Interpreters/ExternalLoader.cpp:1248:36 (clickhouse+0x1572fff7)

  [1]: https://clickhouse-test-reports.s3.yandex.net/29856/42ca2b4bb241827edf69bbd6938d6b19c31935f1/integration_tests_(thread).html#fail1
 2021-10-10 04:04:05 +03:00
Azat Khuzhin
4bc90d1dd7 Fix data-race between flush() and startup() in StorageBuffer 
Stress tests found [1], TSan report:

    ==================
    WARNING: ThreadSanitizer: data race (pid=485)
      Read of size 8 at 0x7b5001280bd8 by thread T567 (mutexes: write M612061890855345680):
        1 std::__1::shared_ptr<DB::BackgroundSchedulePoolTaskInfo>::operator bool() const obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:2851:62 (clickhouse+0x159140a6)
        2 bool std::__1::operator!=<DB::BackgroundSchedulePoolTaskInfo>() obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:3447:30 (clickhouse+0x159140a6)
        3 DB::BackgroundSchedulePoolTaskHolder::operator bool() const obj-x86_64-linux-gnu/../src/Core/BackgroundSchedulePool.h:164:46 (clickhouse+0x159140a6)
        4 DB::StorageBuffer::flush() obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:675:10 (clickhouse+0x159140a6)

      Previous write of size 8 at 0x7b5001280bd8 by thread T586 (mutexes: write M191819750614415520):
        2 std::__1::shared_ptr<DB::BackgroundSchedulePoolTaskInfo>::operator=(std::__1::shared_ptr<DB::BackgroundSchedulePoolTaskInfo>&&) obj-x86_64-linux-gnu/../contrib/libcxx/include/memory:3243:34 (clickhouse+0x15913e22)
        3 DB::BackgroundSchedulePoolTaskHolder::operator=() obj-x86_64-linux-gnu/../src/Core/BackgroundSchedulePool.h:156:110 (clickhouse+0x15913e22)
        4 DB::StorageBuffer::startup() obj-x86_64-linux-gnu/../src/Storages/StorageBuffer.cpp:668:18 (clickhouse+0x15913e22)
        5 DB::InterpreterCreateQuery::doCreateTable() obj-x86_64-linux-gnu/../src/Interpreters/InterpreterCreateQuery.cpp:1092:10 (clickhouse+0x149bef7b)
        6 DB::InterpreterCreateQuery::createTable() obj-x86_64-linux-gnu/../src/Interpreters/InterpreterCreateQuery.cpp:952:20 (clickhouse+0x149ba9f5)
        7 DB::InterpreterCreateQuery::execute() obj-x86_64-linux-gnu/../src/Interpreters/InterpreterCreateQuery.cpp:1302:16 (clickhouse+0x149c1086)

  [1]: https://clickhouse-test-reports.s3.yandex.net/0/1c9778603ff49563d1d3d0d357de0608167e504d/stress_test_(thread).html

Fixes: #29416
 2021-10-10 04:03:36 +03:00
alexey-milovidov
2c3b866ab1
Merge pull request #29931 from azat/fix-build 
Fix build due to conflicts in serverConstants
 2021-10-10 03:56:39 +03:00
Azat Khuzhin
cc1fbe27a7 Fix build due to conflicts in serverConstants 
Refs: #29755
Refs: #29913
 2021-10-10 03:40:51 +03:00
alexey-milovidov
fbbc515a2f
Merge pull request #29925 from ClickHouse/update-cctz-2 
Update CCTZ
 2021-10-10 02:17:09 +03:00
Maksim Kita
84555646ff
Merge pull request #29927 from ClickHouse/better-exception-message-in-reading-arrow 
Better exception message while reading column from Arrow-supported formats
 2021-10-10 01:18:43 +03:00
Maksim Kita
daf9cf12d9
Merge pull request #29755 from Joeywzr/new_func 
add function getOSKernelVersion
 2021-10-10 01:12:54 +03:00
Maksim Kita
b9969232f5
Update 02095_function_get_os_kernel_version.sql 2021-10-10 01:12:36 +03:00
Alexey
17552931af ru other functuins addons 2021-10-09 19:27:24 +00:00
Alexey
2b272f5781 Virtual column in Distributed updated, link fixed, links added 
Translated that part
 2021-10-09 19:17:02 +00:00
Alexey
7f5852a711 New buildId variant 
Links from Distributed
 2021-10-09 18:37:28 +00:00
Alexander Tokmakov
c61f09c4dc fix test for replication consistency 2021-10-09 21:12:15 +03:00
alexey-milovidov
ee577e1ab4
Merge pull request #29155 from FArthur-cmd/add_CORS 
Add Cors support
 2021-10-09 20:18:17 +03:00
Maksim Kita
889034f6c2
Merge pull request #29913 from amosbird/refactor1 
Cosmetic refactoring of server constants.
 2021-10-09 19:59:27 +03:00
Alexey Milovidov
0828ecae97 Better exception message while reading column from Arrow-supported formats 2021-10-09 19:46:36 +03:00
Vladimir C
8ca3c7be3a
Merge pull request #29881 from vesslanjin/optimize-column-string-filter-performance 2021-10-09 19:44:15 +03:00
Nikolai Kochetov
102dd33cd8
Merge pull request #29841 from ClickHouse/coro-example 
Add coroutines example.
 2021-10-09 19:17:30 +03:00
Nikolai Kochetov
f908531b76 Fix debug build. 2021-10-09 19:12:47 +03:00
WangZengrui
bc093e2b23 fix test 2021-10-10 00:06:11 +08:00
alexey-milovidov
f0b7e54e83
Update CHANGELOG.md 2021-10-09 18:42:53 +03:00
Alexey Milovidov
f25fbe3740 Update CCTZ 2021-10-09 18:37:38 +03:00
tavplubix
017b3eccbe
Merge pull request #29901 from ClickHouse/logging_in_zk_client 
Add logging in ZooKeeper client
 2021-10-09 18:25:40 +03:00
alexey-milovidov
268c155b7d
Merge pull request #29856 from azat/clickhouse-test-python-client 
Rewrite clickhouse-test to use python clickhouse_driver
 2021-10-09 17:58:40 +03:00
Artur
c973d6d059 Merge branch 'add_CORS' of https://github.com/FArthur-cmd/ClickHouse into add_CORS 2021-10-09 13:55:56 +00:00
Artur
24568c9de5 update test reference 2021-10-09 13:55:30 +00:00
Azat Khuzhin
c66942bba5 Fix test_input_format_parallel_parsing_memory_tracking::test_memory_tracking_total 
Server has pretty low memory limit 3GB, so let's use local to generate
the data.
 2021-10-09 16:15:42 +03:00
Filatenkov Artur
7753cc9c28
Update users.xml 2021-10-09 16:01:01 +03:00
Filatenkov Artur
98ecd59ebe
Update users.xml 2021-10-09 16:00:37 +03:00