This will reduce amount of data that may be lost at shutdown.
v2: replace dynamic_cast with IStorage::isBuffer (@kitaisreal)
v3: replace IStorage::isBuffer with IStorage::flush (@alexey-milovidov)
v4: flush() for StorageProxy/StorageTableFunction
When Buffer() is under preassure, acquiring per-layer lock may take
significant time. And so the following query may take significant amount of time:
SELECT total_bytes, total_rows FROM system.tables WHERE engine='Buffer'
Add 3 new engine arguments:
- flush_time
- flush_rows
- flush_bytes
That will be checked only for background flush, this maybe useful if
INSERT latency is "crucial".
It uses very fast CLOCK_MONOTONIC_COARSE, so this should not be a
problem.
Also note that there is no sense in using microseconds/nanoseconds since
accuracy of CLOCK_MONOTONIC_COARSE usually milliseconds.
* master: (759 commits)
Suppress UBSan report in Decimal comparison
Suppress UBSan report in Decimal comparison
Fix UBSan report in arrayDifference
Update README.md
Non significant change in AggregationCommon
Print stack trace on SIGTRAP
Fix dependent test
Fix tests for better parallel run
Add test for already working code
Revert "Fix access control manager destruction order"
Update index.md
Update index.md
Update index.md
Bit more complicated example for isIPv4String - ru
Bit more complicated example for isIPv4String
cleanup
Replace database with ordinary
Added comments
Split tests to make them stable
Fixes
...
# Conflicts:
# src/Storages/MergeTree/MergeTreeRangeReader.cpp
If you push data via Buffer engine then all your queries will be done
from one user, however this is not always desired behavior, since this
will not allow to limit queries with max_concurrent_queries_for_user and
similar.
Otherwise you can see something like this for the following query:
```sql
WITH
arrayMap(x -> demangle(addressToSymbol(x)), s.trace) AS trace_array,
arrayStringConcat(trace_array, '\n') AS trace_string
SELECT
p.thread_id,
p.query_id,
p.query,
trace_string
FROM
(
SELECT
query_id,
query,
arrayJoin(thread_ids) AS thread_id
FROM system.processes
) AS p
INNER JOIN system.stack_trace AS s ON p.thread_id = s.thread_id
ORDER BY p.query_id ASC
SETTINGS enable_global_with_statement = 0, allow_introspection_functions = 1
FORMAT PrettyCompactNoEscapes
```
Lots of the following:
```sql
INSERT INTO buffer (...) VALUES
__lll_lock_wait
pthread_mutex_lock
std::__1::mutex::lock()
DB::StorageBuffer::reschedule()
DB::BufferBlockOutputStream::write(DB::Block const&)
```
That will wait one of this:
```
INSERT INTO buffer (...) VALUES
...
DB::PushingToViewsBlockOutputStream::write(DB::Block const&)
DB::AddingDefaultBlockOutputStream::write(DB::Block const&)
DB::SquashingBlockOutputStream::finalize()
DB::SquashingBlockOutputStream::writeSuffix()
DB::PushingToViewsBlockOutputStream::writeSuffix()
DB::StorageBuffer::writeBlockToDestination(DB::Block const&, std::__1::shared_ptr<DB::IStorage>)
DB::StorageBuffer::flushBuffer(DB::StorageBuffer::Buffer&, bool, bool, bool)
```
P.S. we cannot simply unlock the buffer during flushing, see comments in
the code
