Nikolai Kochetov
285cfb7fd5
Fix build and style.
2021-09-10 14:42:52 +03:00
Anton Popov
5cff615eca
Merge remote-tracking branch 'upstream/master' into HEAD
2021-09-10 13:40:48 +03:00
Maksim Kita
21d33c3c2e
ShellCommandSource fix logging
2021-09-10 01:28:22 +03:00
Nikolai Kochetov
f569a3e3f7
Merge branch 'master' into rewrite-pushing-to-views
2021-09-09 20:30:23 +03:00
Nikolai Kochetov
47b96add9a
Refactor some code.
2021-09-09 16:59:15 +03:00
kssenii
1838a8672d
Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into interactive-mode-for-clickhouse-local
2021-09-09 12:43:45 +00:00
Anton Popov
8e5b3b2f6c
Merge remote-tracking branch 'upstream/master' into HEAD
2021-09-09 15:12:34 +03:00
Nikolai Kochetov
aea1871755
Fix some more tests.
2021-09-09 14:16:03 +03:00
Nikolai Kochetov
7a5846c5f7
Comment debug output.
2021-09-08 21:30:54 +03:00
Nikolai Kochetov
999a4fe831
Fix other tests.
2021-09-08 21:29:38 +03:00
Nikolai Kochetov
f34cb097d8
Fix some tests.
2021-09-06 23:13:06 +03:00
Maksim Kita
311110e36b
Merge pull request #28518 from kitaisreal/executable-pool-storage
...
Added ExecutablePool storage
2021-09-04 21:09:07 +03:00
kssenii
1c20b223cb
Merge branch 'master' of github.com:ClickHouse/ClickHouse into interactive-mode-for-clickhouse-local
2021-09-04 20:01:31 +03:00
Anton Popov
59128412ea
Merge remote-tracking branch 'upstream/master' into HEAD
2021-09-03 22:17:11 +03:00
Nikolai Kochetov
66a76ab70f
Rewrite PushingToViewsBlockOutputStream part 6
2021-09-03 20:29:36 +03:00
Maksim Kita
af7220b6aa
Updated ShellCommandSource
2021-09-03 13:00:40 +03:00
Nikolai Kochetov
af0da33147
Merge pull request #27827 from ClickHouse/more-checks-for-lv-and-native
...
Add more checks for LC in native protocol.
2021-09-02 17:27:42 +03:00
Maksim Kita
abda2a636e
Added ExecutablePool storage
2021-09-02 14:53:20 +03:00
Nikolai Kochetov
358babc31b
Rewrite PushingToViewsBlockOutputStream part 5.
2021-09-02 14:29:20 +03:00
Anton Popov
5e421ab272
Merge remote-tracking branch 'upstream/master' into HEAD
2021-09-02 02:42:50 +03:00
Nikolai Kochetov
79ecb6667e
Rewrite PushingToViewsBlockOutputStream part 4.
2021-09-01 21:41:50 +03:00
Alexey Milovidov
19445ac096
Drop unused headers
2021-09-01 02:47:52 +03:00
Nikolai Kochetov
5be05c2ef9
Rewrite PushingToViewsBlockOutputStream part 3.
2021-08-31 16:50:07 +03:00
Maksim Kita
4ebd0ae941
Merge pull request #28102 from ClickHouse/executable-table-function
...
Merging #23192
2021-08-31 12:00:50 +03:00
Nikolai Kochetov
eccee47e72
Rewrite PushingToViewsBlockOutputStream part 2.
2021-08-30 14:03:39 +03:00
Dmitrii Kovalkov
8acf0fa16a
run generate-ya-make
2021-08-30 10:57:40 +03:00
Maksim Kita
4aa2479ca1
Added support for multiple pipes
2021-08-28 22:47:59 +03:00
kssenii
34f9983f14
Merge branch 'master' of github.com:ClickHouse/ClickHouse into interactive-mode-for-clickhouse-local
2021-08-27 09:35:23 +03:00
Kruglov Pavel
12b1e9ae6e
Merge pull request #27881 from azat/remote-query-cancel-fix
...
Fix remote query cancelation (in case of remote server abnormaly terminated)
2021-08-26 16:31:54 +03:00
Nikolai Kochetov
4d821efa15
Rewrite PushingToViewsBlockOutputStream part 2.
2021-08-26 11:01:26 +03:00
Anton Popov
6d3274c22c
Merge remote-tracking branch 'upstream/master' into HEAD
2021-08-26 03:31:46 +03:00
Maksim Kita
6ccdde4a88
Updated TableFunctionExecutable
2021-08-25 22:30:22 +03:00
Azat Khuzhin
e8d87053c0
Fix data-race between setConnectionFD() and setTimer() from cancellation path
...
TSan report [1]:
==================
WARNING: ThreadSanitizer: data race (pid=7)
Write of size 8 at 0x7b5000d83768 by thread T539:
0 Poco::Timespan::operator=(Poco::Timespan const&) obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Timespan.cpp:61:8 (clickhouse+0x18e2da08)
1 DB::RemoteQueryExecutorReadContext::setConnectionFD(int, Poco::Timespan, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) obj-x86_64-linux-gnu/../src/DataStreams/RemoteQueryExecutorReadContext.cpp:103:21 (clickhouse+0x142c4f41)
2 DB::RemoteQueryExecutorRoutine::ReadCallback::operator()(int, Poco::Timespan, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) obj-x86_64-linux-gnu/../src/DataStreams/RemoteQueryExecutorReadContext.cpp:26:30 (clickhouse+0x142c7dae)
3 decltype(std::__1::forward<DB::RemoteQueryExecutorRoutine::ReadCallback&>(fp)(std::__1::forward<int>(fp0), std::__1::forward<Poco::Timespan>(fp0), std::__1::forward<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>(fp0))) std::__1::__invoke<DB::RemoteQueryExecutorRoutine::ReadCallback&, int, Poco::Timespan, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>(DB::RemoteQueryExecutorRoutine::ReadCallback&, int&&, Poco::Timespan&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) obj-x86_64-linux-gnu/../contrib/libcxx/include/type_traits:3676:1 (clickhouse+0x142c7cf7)
4 void std::__1::__invoke_void_return_wrapper<void>::__call<DB::RemoteQueryExecutorRoutine::ReadCallback&, int, Poco::Timespan, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>(DB::RemoteQueryExecutorRoutine::ReadCallback&, int&&, Poco::Timespan&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) obj-x86_64-linux-gnu/../contrib/libcxx/include/__functional_base:348:9 (clickhouse+0x142c7cf7)
5 std::__1::__function::__default_alloc_func<DB::RemoteQueryExecutorRoutine::ReadCallback, void (int, Poco::Timespan, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>::operator()(int&&, Poco::Timespan&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:1608:12 (clickhouse+0x142c7c21)
6 void std::__1::__function::__policy_invoker<void (int, Poco::Timespan, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>::__call_impl<std::__1::__function::__default_alloc_func<DB::RemoteQueryExecutorRoutine::ReadCallback, void (int, Poco::Timespan, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)> >(std::__1::__function::__policy_storage const*, int, Poco::Timespan&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2089:16 (clickhouse+0x142c7c21)
7 std::__1::__function::__policy_func<void (int, Poco::Timespan, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>::operator()(int&&, Poco::Timespan&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2221:16 (clickhouse+0x1589a8c1)
8 std::__1::function<void (int, Poco::Timespan, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>::operator()(int, Poco::Timespan, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const obj-x86_64-linux-gnu/../contrib/libcxx/include/functional:2560:12 (clickhouse+0x1589a8c1)
9 DB::HedgedConnections::getReadyFileDescriptor(std::__1::function<void (int, Poco::Timespan, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>) obj-x86_64-linux-gnu/../src/Client/HedgedConnections.cpp:379:13 (clickhouse+0x1589a8c1)
10 DB::HedgedConnections::getReadyReplicaLocation(std::__1::function<void (int, Poco::Timespan, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>) obj-x86_64-linux-gnu/../src/Client/HedgedConnections.cpp:317:20 (clickhouse+0x1589a8c1)
11 DB::HedgedConnections::receivePacketUnlocked(std::__1::function<void (int, Poco::Timespan, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>, bool) obj-x86_64-linux-gnu/../src/Client/HedgedConnections.cpp:298:32 (clickhouse+0x1589b531)
12 DB::RemoteQueryExecutorRoutine::operator()(boost::context::fiber&&) const obj-x86_64-linux-gnu/../src/DataStreams/RemoteQueryExecutorReadContext.cpp:46:51 (clickhouse+0x142c7013)
13 std::__1::enable_if<!(std::is_member_pointer<std::__1::decay<DB::RemoteQueryExecutorRoutine&>::type>::value), std::__1::result_of<DB::RemoteQueryExecutorRoutine& (boost::context::fiber&&)>::type>::type boost::context::detail::invoke<DB::RemoteQueryExecutorRoutine&, boost::context::fiber>(DB::RemoteQueryExecutorRoutine&, boost::context::fiber&&) obj-x86_64-linux-gnu/../contrib/boost/boost/context/detail/invoke.hpp:41:12 (clickhouse+0x142c6be1)
14 boost::context::detail::fiber_capture_record<boost::context::fiber, FiberStack&, DB::RemoteQueryExecutorRoutine>::run() obj-x86_64-linux-gnu/../contrib/boost/boost/context/fiber_ucontext.hpp:291:17 (clickhouse+0x142c6be1)
15 void boost::context::detail::fiber_entry_func<boost::context::detail::fiber_capture_record<boost::context::fiber, FiberStack&, DB::RemoteQueryExecutorRoutine> >(void*) obj-x86_64-linux-gnu/../contrib/boost/boost/context/fiber_ucontext.hpp:72:13 (clickhouse+0x142c5f79)
16 <null> <null> (libc.so.6+0x5e65f)
Previous read of size 8 at 0x7b5000d83768 by thread T321 (mutexes: write M861449115349678976, write M1059607378698108960, write M747733173224015304):
0 Poco::Timespan::totalMicroseconds() const obj-x86_64-linux-gnu/../contrib/poco/Foundation/include/Poco/Timespan.h:213:9 (clickhouse+0x142c54b3)
1 DB::RemoteQueryExecutorReadContext::setTimer() const obj-x86_64-linux-gnu/../src/DataStreams/RemoteQueryExecutorReadContext.cpp:160:25 (clickhouse+0x142c54b3)
2 DB::RemoteQueryExecutor::tryCancel(char const*, std::__1::unique_ptr<DB::RemoteQueryExecutorReadContext, std::__1::default_delete<DB::RemoteQueryExecutorReadContext> >*) obj-x86_64-linux-gnu/../src/DataStreams/RemoteQueryExecutor.cpp:538:26 (clickhouse+0x142be3a5)
3 DB::RemoteQueryExecutor::cancel(std::__1::unique_ptr<DB::RemoteQueryExecutorReadContext, std::__1::default_delete<DB::RemoteQueryExecutorReadContext> >*) obj-x86_64-linux-gnu/../src/DataStreams/RemoteQueryExecutor.cpp:462:5 (clickhouse+0x142bdd8a)
4 DB::RemoteSource::onCancel() obj-x86_64-linux-gnu/../src/Processors/Sources/RemoteSource.cpp:112:21 (clickhouse+0x15bf52c6)
5 DB::IProcessor::cancel() obj-x86_64-linux-gnu/../src/Processors/IProcessor.h:237:9 (clickhouse+0x159db37e)
6 DB::PipelineExecutor::cancel() obj-x86_64-linux-gnu/../src/Processors/Executors/PipelineExecutor.cpp:376:20 (clickhouse+0x159db37e)
7 DB::PullingAsyncPipelineExecutor::cancel() obj-x86_64-linux-gnu/../src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:175:25 (clickhouse+0x159e9748)
8 DB::TCPHandler::processOrdinaryQueryWithProcessors() obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:700:26 (clickhouse+0x1594f554)
9 DB::TCPHandler::runImpl() obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:329:17 (clickhouse+0x15949a2e)
10 DB::TCPHandler::run() obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:1658:9 (clickhouse+0x15959007)
11 Poco::Net::TCPServerConnection::start() obj-x86_64-linux-gnu/../contrib/poco/Net/src/TCPServerConnection.cpp:43:3 (clickhouse+0x18cb2e42)
12 Poco::Net::TCPServerDispatcher::run() obj-x86_64-linux-gnu/../contrib/poco/Net/src/TCPServerDispatcher.cpp:115:20 (clickhouse+0x18cb35cf)
13 Poco::PooledThread::run() obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/ThreadPool.cpp:199:14 (clickhouse+0x18e29c21)
14 Poco::(anonymous namespace)::RunnableHolder::run() obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Thread.cpp:55:11 (clickhouse+0x18e280af)
15 Poco::ThreadImpl::runnableEntry(void*) obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Thread_POSIX.cpp:345:27 (clickhouse+0x18e26827)
[1]: https://clickhouse-test-reports.s3.yandex.net/27881/b328d9687238617e89c0806c9fa2dd7dd44b5fd7/integration_tests_(thread).htmlfail1
2021-08-25 09:42:30 +03:00
Azat Khuzhin
225f44248c
Fix remote query cancelation (in case of remote server abnormaly terminated)
...
In case the remote server abnormally terminated, neither FIN nor RST
packet will be sent, and the initiator will not know that the connection
died (unless tcp_keep_alive_timeout > 0).
Fix this, using default timeouts.
2021-08-25 09:42:30 +03:00
lthaooo
314ba1e682
Merge branch 'master' into materialize_ttl_recalculate_only
2021-08-25 10:56:24 +08:00
Maksim Kita
5545959581
Updated StorageExecutable
2021-08-24 22:38:42 +03:00
Maksim Kita
016c7c74f8
Merge branch 'executable-table-function' into add-executable-table-function
2021-08-24 21:45:43 +03:00
mergify[bot]
ae22bab2c4
Merge branch 'master' into materialize_ttl_recalculate_only
2021-08-24 08:49:19 +00:00
Nikolai Kochetov
d7e78f3ea9
Rewrite PushingToViewsBlockOutputStream part 1.
2021-08-23 13:46:52 +03:00
Vitaly Baranov
ff1fad8521
Use setSetting() with implicit conversion true/false -> Field.
2021-08-23 08:57:00 +03:00
Nikolai Kochetov
108137a67e
Add more checks for LC in native protocol.
2021-08-18 19:15:57 +03:00
kssenii
fb0b445ee0
Lets add LocalConnection (a start)
2021-08-17 22:59:51 +03:00
Maksim Kita
03fcaf6149
Merge pull request #27639 from Algunenano/better_mv_logs_parallel
...
MV: Improve text logs when doing parallel processing
2021-08-13 19:42:25 +03:00
Nikolai Kochetov
6b1030c9b8
Rewrite PushingToViewsBIS part 1.
2021-08-13 19:32:29 +03:00
Raúl Marín
7fba508b00
MV: Improve text logs when doing parallel processing
2021-08-13 13:43:37 +02:00
Nikolai Kochetov
ad00aaa18c
Merge pull request #27575 from kitaisreal/removed-some-data-streams
...
Removed some data streams
2021-08-13 12:59:00 +03:00
Maksim Kita
d9a59370d3
Fixed SQLiteSource
2021-08-13 00:58:24 +03:00
mergify[bot]
21cbd50ae1
Merge branch 'master' into materialization_log
2021-08-12 08:53:34 +00:00
Maksim Kita
124a87684f
Removed some data streams
2021-08-11 23:39:01 +03:00
alexey-milovidov
33ad51588d
Merge pull request #27569 from filimonov/less_include_stopwatch
...
Less Stopwatch.h
2021-08-11 22:32:24 +03:00
Mikhail Filimonov
af6249c8ec
Less include <Common/Stopwatch.h>
2021-08-11 16:20:59 +02:00
mergify[bot]
f590cc9ef4
Merge branch 'master' into materialization_log
2021-08-11 13:34:00 +00:00
kssenii
eaea0a3e40
fix
2021-08-11 12:05:38 +00:00
Raúl Marín
7324c7cd6b
Style #2
2021-08-10 15:35:39 +02:00
Maksim Kita
87fb75b272
Merge pull request #27191 from FArthur-cmd/memory-tracker-26043
...
Client with flag for memory tracker
2021-08-10 14:01:02 +03:00
Raúl Marín
b28a186727
Style
2021-08-10 12:08:46 +02:00
Raúl Marín
315b611963
MV: Use as many threads as views if max_threads = 0
2021-08-10 11:51:32 +02:00
Raúl Marín
820a6e0987
Refactor MV stage metric setup and teardown
2021-08-10 10:49:12 +02:00
Raúl Marín
d1d47658f5
Merge remote-tracking branch 'blessed/master' into materialization_log
2021-08-10 10:49:05 +02:00
Nikolai Kochetov
b5d3bc34d2
Try fix integration tests.
2021-08-09 16:27:19 +03:00
Maksim Kita
b8b24684b4
Merge pull request #27273 from ClickHouse/remove-streams-from-dicts
...
Remove streams from dicts
2021-08-09 15:58:37 +03:00
Mikhail Filimonov
31a9b6d5ca
Less clock_gettime calls
2021-08-06 17:38:55 +02:00
Anton Popov
3a0d4807a5
Merge branch 'async-insert' of git://github.com/abyss7/ClickHouse into merging-20557
2021-08-06 16:20:02 +03:00
Nikolai Kochetov
39ff5ab478
Merge branch 'master' into remove-streams-from-dicts
2021-08-06 11:46:27 +03:00
Nikolai Kochetov
13f95f3fdf
Streams -> Processors for dicts, part 3.
2021-08-06 11:41:45 +03:00
Nikolai Kochetov
8546df13c2
Streams -> Processors for dicts, part 2.
2021-08-05 21:08:52 +03:00
Artur
d2869d2e2b
add test for memory tracker in client
2021-08-05 16:20:24 +00:00
Raúl Marín
479f053f2e
Merge remote-tracking branch 'blessed/master' into materialization_log
2021-08-05 17:24:10 +02:00
terrylin
88b9e7f24f
materialize ttl recalculate only optional
2021-08-04 22:16:13 +08:00
terrylin
ffe9eab1d0
update test cases and fix bug
2021-08-03 15:37:25 +08:00
Raúl Marín
9f32ecca89
Merge remote-tracking branch 'blessed/master' into materialization_log
2021-08-02 16:21:10 +02:00
Anton Popov
cc3ed12ed6
Merge remote-tracking branch 'upstream/master' into HEAD
2021-08-02 02:56:00 +03:00
terrylin
09a4f82d0f
code style check
2021-07-31 20:36:40 +08:00
terrylin
e740f5e6d5
fix compile error
2021-07-30 20:48:50 +08:00
terrylin
c49c2e5e18
fix compile error
2021-07-30 17:15:04 +08:00
alexey-milovidov
407f53b476
Merge pull request #26953 from ClickHouse/fix-race-in-connections-collector
...
Hold context in HedgedConnections to prevent use-after-free on settings.
2021-07-30 08:52:17 +03:00
terrylin
55f3e7e65a
improvement of materilize ttl
2021-07-30 12:27:15 +08:00
Raúl Marín
34b0bb6b7f
Move formatBlock to its own file
2021-07-29 16:39:42 +02:00
Nikolai Kochetov
cfa3fd4b87
Hold context in HedgedConnections to prevent use-after-free on settings.
2021-07-29 13:29:36 +03:00
Raúl Marín
4daf3f61ea
Rebased with the latest changes in master
2021-07-27 16:46:10 +02:00
Raúl Marín
7e27b773d3
MV: Don't play with ThreadStatus if we aren't inside a query
2021-07-27 16:29:34 +02:00
Raúl Marín
457c6e6b79
MV: Report memory usage when outside of a query
2021-07-27 16:29:34 +02:00
Raúl Marín
fef9870c82
Revert clang-format style change and see if that makes CI happy
2021-07-27 16:29:34 +02:00
Raúl Marín
db0cdd696f
Dependent views: Report progress
2021-07-27 16:29:34 +02:00
Raúl Marín
0d9bf0a08e
Avoid havin multiple profilers in the same thread
2021-07-27 16:29:34 +02:00
Raúl Marín
104bd9cde0
Views: Improve scope handling
2021-07-27 16:29:34 +02:00
Raúl Marín
0231583ea2
PushingToViewsBlockOutputStream: Use setupState instead
2021-07-27 16:29:34 +02:00
Raúl Marín
e4b65da6e4
PushingToViewsBlockOutputStream: Detatch ThreadStatus'
2021-07-27 16:29:34 +02:00
Raúl Marín
75d18bc79b
CI linters and compiler errors
2021-07-27 16:29:34 +02:00
Raúl Marín
69577a782f
Views: Once started, finish all concurrent views before throwing
...
Since sequential processing executes things in phases too, try to finish
all (non errored out) views before throwing any exception
2021-07-27 16:29:33 +02:00
Raúl Marín
ce501bcf25
MatViews: Attach memory trackers to current query
2021-07-27 16:29:33 +02:00
Raúl Marín
70b7bb4eaa
Refresh table acecss information before the final query_log
...
Table access should show the tables used by joins in dependent views
2021-07-27 16:29:33 +02:00
Raúl Marín
f0125262b7
Add views and it's tables to system.query_log
2021-07-27 16:29:33 +02:00
Raúl Marín
e17af6113b
Fix live views
2021-07-27 16:29:33 +02:00
Raúl Marín
fc23e212c2
ThreadStatus: Don't change current thread if it's not this
2021-07-27 16:29:33 +02:00
Raúl Marín
eb994d8f04
Fix a couple of bugs
...
- Don't use threads if there is only one dependent view and parallel_view
- Reset the current_thread when destroying views ThreadStatus
2021-07-27 16:29:33 +02:00
Raúl Marín
7c1fcc94b0
View log: Make sure to only profile events during view execution
2021-07-27 16:29:33 +02:00
Raúl Marín
e0ee7839f7
Only use threads when necessary and log written status
2021-07-27 16:29:31 +02:00
Raúl Marín
6b9ec2a62e
WIP
2021-07-27 16:28:28 +02:00
Raúl Marín
efe1938437
WIP: Slow improvements
2021-07-27 16:26:29 +02:00