Commit Graph

89894 Commits

Author SHA1 Message Date
kssenii
69cd3a2b10 Fix 2022-05-31 14:20:31 +02:00
Alexander Tokmakov
30a7b07d97
Merge pull request #37658 from vitlibar/fix-flaky-test_row_policy
Fix flaky test test_row_policy
2022-05-31 12:59:50 +03:00
alesapin
65057bf8c4
Merge pull request #37616 from ClickHouse/remove-resursive-submodules
Remove resursive submodules
2022-05-31 11:58:04 +02:00
Kseniia Sumarokova
73ed9c3977
Merge pull request #37619 from Vxider/wv-fix-table-identifier
Fix bugs in WindowView when using table identifier
2022-05-31 11:07:11 +02:00
Robert Schulze
32c810fd35
Merge pull request #37644 from ClickHouse/fix-amqp-cpp-cassandra-dependencies
Disable amqp-cpp and cassandra build if libuv is disabled
2022-05-31 10:47:38 +02:00
Robert Schulze
557bb2d235
Disable amqp-cpp and cassandra build if libuv is disabled
On MacOS/GCC, the libuv build is disabled due to a compiler bug. This
is now propagated to dependent libraries amqp-cpp and cassandra.
Oddly enough, the Mac/GCC build was broken since at least Jan 2022
without someone noticing.
2022-05-31 10:34:03 +02:00
Yakov Olkhovskiy
873ac9f8ff
Merge pull request #37540 from ClickHouse/feature-server-certificate
showCertificate function implementation
2022-05-31 02:50:03 -04:00
Yakov Olkhovskiy
c6b20cd5ed
Merge pull request #37187 from Algunenano/floating_seconds
Allow decimal values in settings using seconds
2022-05-30 20:33:47 -04:00
Kseniia Sumarokova
18bda56e4c
Merge pull request #37655 from ClickHouse/kssenii-patch-3-1
Fix hung check
2022-05-30 21:22:12 +02:00
mergify[bot]
b43cfd056f
Merge branch 'master' into floating_seconds 2022-05-30 19:18:35 +00:00
Nikolai Kochetov
77b07dd0a8
Merge pull request #37163 from ClickHouse/grouping-function
Add GROUPING function
2022-05-30 20:45:04 +02:00
Anton Popov
52d3791eb9
Merge pull request #37600 from CurtizJ/fix-with-fill-interval
Fix `WITH FILL` with negative intervals in `STEP` clause
2022-05-30 19:43:12 +02:00
alesapin
6db44f633f
Merge pull request #37641 from azat/keeper-list-watches
keeper: store only unique session IDs for watches (should fix SIGKILL in stress tests)
2022-05-30 18:55:52 +02:00
Vitaly Baranov
486a11a5e2 Fix flaky test test_row_policy. 2022-05-30 18:34:28 +02:00
Kseniia Sumarokova
b1ba7b7027
Fix build 2022-05-30 17:30:59 +02:00
Kseniia Sumarokova
1869adfd7d
Update FileCache.cpp 2022-05-30 16:06:58 +02:00
Alexander Tokmakov
351956d108
Merge pull request #37640 from azat/transaction-fix
Fix excessive LIST requests to coordinator for transactions
2022-05-30 15:46:52 +03:00
alesapin
84ed5aa6b0 No recursive in CI 2022-05-30 14:41:27 +02:00
Kruglov Pavel
0615866aea
Merge pull request #37450 from Avogar/check-format-on-storage-creation
Check format name on storage creation
2022-05-30 14:23:20 +02:00
Maksim Kita
55f1faf77d
Merge pull request #37637 from ucasfl/typo
fix typo
2022-05-30 11:03:35 +02:00
Azat Khuzhin
d86181d3cd keeper: store only unique session IDs for watches
This should speed up keeper, especially in case of incorrect usage (like
the case that had been fixed in #37640), especially in case on non
release build.

And also this should fix SIGKILL in stress tests.

You will find some details for one of such SIGKILL in `<details>` tag [1]:

<details>

    $ pigz -cd clickhouse-server.stress.log.gz | tail
    2022.05.27 16:17:24.882971 [ 637 ] {} <Trace> BackgroundSchedulePool/BgSchPool: Waiting for threads to finish.
    2022.05.27 16:17:24.896749 [ 637 ] {} <Debug> MemoryTracker: Peak memory usage (for query): 4.09 MiB.
    2022.05.27 16:17:24.907163 [ 637 ] {} <Debug> Application: Shut down storages.
    2022.05.27 16:17:24.907233 [ 637 ] {} <Debug> Application: Waiting for current connections to servers for tables to finish.
    2022.05.27 16:17:24.934335 [ 637 ] {} <Information> Application: Closed all listening sockets. Waiting for 1 outstanding connections.
    2022.05.27 16:17:29.843491 [ 637 ] {} <Information> Application: Closed connections to servers for tables. But 1 remain. Probably some tables of other users cannot finish their connections after context shutdown.
    2022.05.27 16:17:29.843632 [ 637 ] {} <Debug> KeeperDispatcher: Shutting down storage dispatcher
    2022.05.27 16:17:34.612616 [ 688 ] {} <Test> virtual Coordination::ZooKeeperRequest::~ZooKeeperRequest(): Processing of request xid=2147483647 took 10000 ms
    2022.05.27 16:17:54.612109 [ 3176 ] {} <Debug> KeeperTCPHandler: Session #12 expired
    2022.05.27 16:19:59.823038 [ 635 ] {} <Fatal> Application: Child process was terminated by signal 9 (KILL). If it is not done by 'forcestop' command or manually, the possible cause is OOM Killer (see 'dmesg' and look at the '/var/log/kern.log' for the details).

    Thread 26 (Thread 0x7f1c7703f700 (LWP 708)):
    0  0x000000000b074b2a in __tsan::MemoryAccessImpl(__tsan::ThreadState*, unsigned long, int, bool, bool, unsigned long long*, __tsan::Shadow) ()
    1  0x000000000b08630c in __tsan::MemoryAccessRange(__tsan::ThreadState*, unsigned long, unsigned long, unsigned long, bool) ()
    2  0x000000000b01ff03 in memmove ()
    3  0x000000001bbc8996 in std::__1::__move<long, long> (__first=0xb8600000d83304, __last=<optimized out>, __result=0x7f1c021cd000) at ../contrib/libcxx/include/__algorithm/move.h:57
    4  std::__1::move<long*, long*> (__first=0xb8600000d83304, __last=<optimized out>, __result=0x7f1c021cd000) at ../contrib/libcxx/include/__algorithm/move.h:70
    5  std::__1::vector<long, std::__1::allocator<long> >::erase (this=0x7b1400584c48, __position=...) at ../contrib/libcxx/include/vector:1608
    6  DB::KeeperStorage::clearDeadWatches (this=0x7b5800001ad8, this@entry=0x7b5800001800, session_id=session_id@entry=12) at ../src/Coordination/KeeperStorage.cpp:1228
    7  0x000000001bbc5c55 in DB::KeeperStorage::processRequest (this=0x7b5800001800, zk_request=..., session_id=12, time=1, new_last_zxid=..., check_acl=true) at ../src/Coordination/KeeperStorage.cpp:1122
    8  0x000000001bba06a3 in DB::KeeperStateMachine::commit (this=<optimized out>, log_idx=3549, data=...) at ../src/Coordination/KeeperStateMachine.cpp:143
    9  0x000000001bba6193 in nuraft::state_machine::commit_ext (this=0x7b4c00001f98, params=...) at ../contrib/NuRaft/include/libnuraft/state_machine.hxx:75
    10 0x00000000202c5a55 in nuraft::raft_server::commit_app_log (this=this@entry=0x7b6c00002a18, idx_to_commit=idx_to_commit@entry=3549, le=..., need_to_handle_commit_elem=true, initial_commit_exec=false) at ../contrib/NuRaft/src/handle_commit.cxx:311
    11 0x00000000202c4f98 in nuraft::raft_server::commit_in_bg_exec (this=<optimized out>, this@entry=0x7b6c00002a18, timeout_ms=timeout_ms@entry=0, initial_commit_exec=false) at ../contrib/NuRaft/src/handle_commit.cxx:241
    12 0x00000000202c4613 in nuraft::raft_server::commit_in_bg (this=this@entry=0x7b6c00002a18) at ../contrib/NuRaft/src/handle_commit.cxx:149
    ...
    Thread 28 (Thread 0x7f1c7603d700 (LWP 710)):
    0  0x00007f1d22a6d110 in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
    1  0x00007f1d22a650a3 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
    2  0x000000000b0337b0 in pthread_mutex_lock ()
    3  0x00000000221884da in std::__1::__libcpp_mutex_lock (__m=0x7b4c00002088) at ../contrib/libcxx/include/__threading_support:303
    4  std::__1::mutex::lock (this=0x7b4c00002088) at ../contrib/libcxx/src/mutex.cpp:33
    5  0x000000001bba4188 in std::__1::lock_guard<std::__1::mutex>::lock_guard (__m=..., this=<optimized out>) at ../contrib/libcxx/include/__mutex_base:91
    6  DB::KeeperStateMachine::getDeadSessions (this=0x7b4c00001f98) at ../src/Coordination/KeeperStateMachine.cpp:360
    7  0x000000001bb79b4b in DB::KeeperServer::getDeadSessions (this=0x7b4400012700) at ../src/Coordination/KeeperServer.cpp:572
    8  0x000000001bb64d1a in DB::KeeperDispatcher::sessionCleanerTask (this=<optimized out>, this@entry=0x7b640001c218) at ../src/Coordination/KeeperDispatcher.cpp:399
    ...
    Thread 1 (Thread 0x7f1d227148c0 (LWP 637)):
    0  0x00007f1d22a69376 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
    1  0x000000000b0895e0 in __tsan::call_pthread_cancel_with_cleanup(int (*)(void*), void (*)(void*), void*) ()
    2  0x000000000b017091 in pthread_cond_wait ()
    3  0x0000000020569d98 in Poco::EventImpl::waitImpl (this=0x7b2000008798) at ../contrib/poco/Foundation/src/Event_POSIX.cpp:106
    4  0x000000001bb636cf in Poco::Event::wait (this=0x7b2000008798) at ../contrib/poco/Foundation/include/Poco/Event.h:97
    5  ThreadFromGlobalPool::join (this=<optimized out>) at ../src/Common/ThreadPool.h:217
    6  DB::KeeperDispatcher::shutdown (this=0x7b640001c218) at ../src/Coordination/KeeperDispatcher.cpp:322
    7  0x0000000019ca8bfc in DB::Context::shutdownKeeperDispatcher (this=<optimized out>) at ../src/Interpreters/Context.cpp:2111
    8  0x000000000b0a979b in DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_9::operator()() const (this=0x7ffcde44f0a0) at ../programs/server/Server.cpp:1407

</details>

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/37593/2613149f6bf4f242bbbf2c3c8539b5176fd77286/stress_test__thread__actions_.html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-05-30 11:50:48 +03:00
Azat Khuzhin
78bd47d8df Fix excessive LIST requests to coordinator for transactions
In [1] there was only few transactions, but lots of List for /test/clickhouse/txn/log:

    $ clickhouse-local --format TSVWithNamesAndTypes --file zookeeper_log.tsv.gz -q "select * except('path|session_id|event_time|thread_id|event_date|xid') apply(x->groupUniqArray(x)), path, session_id, min(event_time), max(event_time), count() from table where has_watch and type = 'Request' group by path, session_id order by count() desc limit 1 format Vertical"
    Row 1:
    ──────
    groupUniqArray(type):             ['Request']
    groupUniqArray(query_id):         ['','62d75128-9031-48a5-87ba-aec3f0b591c6']
    groupUniqArray(address):          ['::1']
    groupUniqArray(port):             [9181]
    groupUniqArray(has_watch):        [1]
    groupUniqArray(op_num):           ['List']
    groupUniqArray(data):             ['']
    groupUniqArray(is_ephemeral):     [0]
    groupUniqArray(is_sequential):    [0]
    groupUniqArray(version):          []
    groupUniqArray(requests_size):    [0]
    groupUniqArray(request_idx):      [0]
    groupUniqArray(error):            []
    groupUniqArray(watch_type):       []
    groupUniqArray(watch_state):      []
    groupUniqArray(stat_version):     [0]
    groupUniqArray(stat_cversion):    [0]
    groupUniqArray(stat_dataLength):  [0]
    groupUniqArray(stat_numChildren): [0]
    groupUniqArray(children):         [[]]
    path:                             /test/clickhouse/txn/log
    session_id:                       1
    min(event_time):                  2022-05-27 12:54:09.025897
    max(event_time):                  2022-05-27 13:37:12.846314 <!-- last transaction was at 12:54, see server log
    count():                          3673675 <-- huge

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/37593/2613149f6bf4f242bbbf2c3c8539b5176fd77286/stateless_tests__debug__actions__[1/3].html

Server log:

    $ pigz -cd clickhouse-server.log.gz | fgrep TransactionLog: | head -1
    2022.05.27 12:54:09.026852 [ 5018 ] {62d75128-9031-48a5-87ba-aec3f0b591c6} <Trace> TransactionLog: Loading 33 entries from /test/clickhouse/txn/log: csn-0000000000..csn-0000000032
    $ pigz -cd clickhouse-server.log.gz | fgrep TransactionLog: | tail -1
    2022.05.27 12:54:58.909222 [ 509 ] {} <Test> TransactionLog: Closing readonly transaction (177, 38, 41b51ff1-bcba-43bf-bcea-e97ad05f6040)
    $ pigz -cd clickhouse-server.log.gz | fgrep 62d75128-9031-48a5-87ba-aec3f0b591c6 | tail -1
    2022.05.27 12:54:09.064857 [ 5018 ] {62d75128-9031-48a5-87ba-aec3f0b591c6} <Debug> MemoryTracker: Peak memory usage (for query): 0.00 B.

Fixes: #37398 (cc @tavplubix)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-05-30 11:39:34 +03:00
flynn
9ffa5f5e0d fix typo 2022-05-30 08:10:16 +00:00
Alexey Milovidov
33b2a50740
Merge pull request #37630 from wangqinghuan/add-clickcat
add clickcat intro
2022-05-30 04:25:22 +03:00
qinghuan wang
ded9f5675d
Update gui.md 2022-05-30 09:22:06 +08:00
Alexey Milovidov
746ff42239
Update gui.md 2022-05-30 01:13:03 +03:00
alesapin
aecab57e17
Merge pull request #37629 from ClickHouse/fix_failure
Fix refactoring issue
2022-05-29 18:35:54 +02:00
qinghuan wang
c14b547f00
Update gui.md 2022-05-29 20:19:23 +08:00
alesapin
d2cdbf3956 Fix refactoring issue 2022-05-29 14:09:49 +02:00
alesapin
9dc81e5cc8
Merge pull request #37598 from ClickHouse/revert-37545-revert-37424-fix_fetching_part_deadlock
Revert "Revert "(only with zero-copy replication, non-production experimental feature not recommended to use) fix possible deadlock during fetching part""
2022-05-29 13:54:34 +02:00
Alexey Milovidov
6195429020
Update star-schema.md 2022-05-29 10:01:27 +03:00
Alexey Milovidov
c8935d6e6c
Update star-schema.md 2022-05-29 10:00:34 +03:00
Alexey Milovidov
e22a79eaf2
Update star-schema.md 2022-05-29 09:58:54 +03:00
Alexander Tokmakov
579b0e3323
Merge pull request #37627 from ClickHouse/revert-37416-fix_ReplicatedMergeTree_comments
Revert "Implemented changing comment to a ReplicatedMergeTree table"
2022-05-29 09:57:12 +03:00
Alexey Milovidov
9e3242f186
Merge pull request #37617 from CurtizJ/aggregation-sparse-columns
Better performance with sparse columns in aggregate functions
2022-05-29 09:36:07 +03:00
Alexander Tokmakov
562eec591e
Revert "Implemented changing comment to a ReplicatedMergeTree table" 2022-05-29 09:28:47 +03:00
Alexey Milovidov
97606c324c
Merge pull request #37574 from azat/mt-tiny-refactor
Remove unused MergeTreeDataMergerMutator::chooseMergeAlgorithm()
2022-05-29 07:59:57 +03:00
Alexey Milovidov
975f16049e
Merge pull request #37597 from azat/fix-01317_no_password_in_command_line
tests: fix 01317_no_password_in_command_line flakiness (and make it race free)
2022-05-29 07:59:10 +03:00
Alexey Milovidov
9e0f61a74f
Merge pull request #37623 from den-crane/patch-12
Doc. Notes about UInt64 key for dictionaries layout
2022-05-29 07:58:07 +03:00
Alexey Milovidov
118b577607
Merge pull request #37626 from ClickHouse/alexey-milovidov-patch-4
Update table structure for "ontime" dataset
2022-05-29 07:42:27 +03:00
Alexey Milovidov
be586f7b33
Update ontime.md 2022-05-29 06:43:50 +03:00
Alexey Milovidov
73c89e65e6
Update ontime.md 2022-05-29 06:31:15 +03:00
Alexey Milovidov
7be278c9ff
Update table structure for "ontime" dataset 2022-05-29 06:26:13 +03:00
Alexey Milovidov
73e2e63414
Merge pull request #37612 from ClickHouse/clang-tidy-14
Fix clang-tidy-14, part 1
2022-05-29 03:16:32 +03:00
Alexey Milovidov
b2ba2533b1
Merge pull request #35231 from ClickHouse/time-zone-in-stateless-tests
Change timezone in stateless tests
2022-05-29 03:09:12 +03:00
Denny Crane
d7e098fb17
Update external-dicts-dict-layout.md 2022-05-28 17:26:11 -03:00
Denny Crane
1d2cf73b81
Update external-dicts-dict-layout.md 2022-05-28 17:14:36 -03:00
Denny Crane
2395764a92
Update external-dicts-dict-layout.md 2022-05-28 17:07:39 -03:00
Denny Crane
bfc7cbaf1f
Update external-dicts-dict-layout.md 2022-05-28 17:02:27 -03:00
Denny Crane
4573c2431e
Update external-dicts-dict-layout.md 2022-05-28 16:59:55 -03:00