SELECT from Buffer table is racy, so you can get data from the
underlying table but not from the Buffer itself, since in parallel with
SELECT, Buffer, can flush it's data to the underlying table.
It is hard to avoid with the current architecture, since this will
require to holding lock until the data will be read from the Buffer, and
this is not a good alternative.
So let's fix the test instead, but not relying on background flush (TTL
increased).
Here is an example of a test failure [1]:
2022.03.12 20:56:58.141182 [ 678 ] {011e7d25-82a9-4ab6-8cb0-dcbbc84f9581} <Debug> executeQuery: (from [::1]:33324) (comment: 01506_buffer_table_alter_block_structure_2.sql) SELECT * FROM buf ORDER BY timestamp;
2022.03.12 20:56:58.162709 [ 678 ] {011e7d25-82a9-4ab6-8cb0-dcbbc84f9581} <Trace> MergeTreeInOrderSelectProcessor: Reading 1 ranges in order from part 20200101_1_1_0, approx. 1 rows starting from 0
2022.03.12 20:56:59.144663 [ 615 ] {} <Trace> test_bdtzgu.buf_dest (79ba36b2-0e90-4bbb-b55f-a42b605b362b): Renaming temporary part tmp_insert_20200101_2_2_0 to 20200101_2_2_0.
2022.03.12 20:56:59.147550 [ 615 ] {} <Debug> StorageBuffer (test_bdtzgu.buf): Flushing buffer with 1 rows, 18 bytes, age 1 seconds, took 19 ms (bg).
2022.03.12 20:56:59.391774 [ 678 ] {011e7d25-82a9-4ab6-8cb0-dcbbc84f9581} <Information> executeQuery: Read 1 rows, 13.00 B in 1.250102785 sec., 0 rows/sec., 10.40 B/sec.
[1]: https://s3.amazonaws.com/clickhouse-test-reports/0/044cd6b861c1f4f00c6c24c4020799b676de6d34/stateless_tests__memory__actions__[1/3].html
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
* Fix unexpected result when use -state type aggregate function in window frame
* fix style
* fix style
* fix test
* fix flaky test
* fix flaky test
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
In case of Buffer table has columns of AggregateFunction type,
aggregate states for such columns will be allocated from the query
context but those states can be destroyed from the server context (in
case of background flush), and thus memory will be leaked from the query
since aggregate states can be shared, and eventually this will lead to
MEMORY_LIMIT_EXCEEDED error.
To avoid this, prohibit sharing the aggregate states.
But note, that this problem only about memory accounting, not memory
usage itself.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
For async s3 writes final part flushing was defered until all the INSERT
block was processed, however in case of too many partitions/columns you
may exceed max_memory_usage limit (since each stream has overhead).
Introduce max_insert_delayed_streams_for_parallel_writes (with default
to 1000 for S3, 0 otherwise), to avoid this.
This should "Memory limit exceeded" errors in performance tests.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
* Use INITIAL_QUERY for clickhouse-benchmark
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
* Fix parallel_reading_from_replicas with clickhouse-bechmark
Before it produces the following error:
$ clickhouse-benchmark --stacktrace -i1 --query "select * from remote('127.1', default.data_mt) limit 10" --allow_experimental_parallel_reading_from_replicas=1 --max_parallel_replicas=3
Loaded 1 queries.
Logical error: 'Coordinator for parallel reading from replicas is not initialized'.
Aborted (core dumped)
Since it uses the same code, i.e RemoteQueryExecutor ->
MultiplexedConnections, which enables coordinator if it was requested
from settings, but it should be done only for non-initial queries, i.e.
when server send connection to another server.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
* Fix 02226_parallel_reading_from_replicas_benchmark for older shellcheck
By shellcheck 0.8 does not complains, while on CI shellcheck 0.7.0 and
it does complains [1]:
In 02226_parallel_reading_from_replicas_benchmark.sh line 17:
--allow_experimental_parallel_reading_from_replicas=1
^-- SC2191: The = here is literal. To assign by index, use ( [index]=value ) with no spaces. To keep as literal, quote it.
Did you mean:
"--allow_experimental_parallel_reading_from_replicas=1"
[1]: https://s3.amazonaws.com/clickhouse-test-reports/34751/d883af711822faf294c876b017cbf745b1cda1b3/style_check__actions_/shellcheck_output.txt
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
* Add a warning if parallel_distributed_insert_select was ignored
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
* Respect max_distributed_depth for parallel_distributed_insert_select
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
* Print warning for non applied parallel_distributed_insert_select only for initial query
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
* Remove Cluster::getHashOfAddresses()
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
* Forbid parallel_distributed_insert_select for remote()/cluster() with different addresses
Before it uses empty cluster name (getClusterName()) which is not
correct, compare all addresses instead.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
* Fix max_distributed_depth check
max_distributed_depth=1 must mean not more then one distributed query,
not two, since max_distributed_depth=0 means no limit, and
distribute_depth is 0 for the first query.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
* Fix INSERT INTO remote()/cluster() with parallel_distributed_insert_select
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
* Add a test for parallel_distributed_insert_select with cluster()/remote()
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
* Return <remote> instead of empty cluster name in Distributed engine
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
* Make user with sharding_key and w/o in remote()/cluster() identical
Before with sharding_key the user was "default", while w/o it it was
empty.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
* Add columns description to metadata in case of schema inference
* Make better
* Remove unnecessary code
* Fix tests
* More tests
* Add tag no-fasttest
* Fix test
* Fix test
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
