Commit Graph

5 Commits

Author SHA1 Message Date
Azat Khuzhin
837f4ea676 Add ability to throttle merges/mutations
Main motivation was to has an ability to throttle background tasks, to
avoid affecting queries.

To new server settings had been added for this:
- max_mutations_bandwidth_for_server
- max_merges_bandwidth_for_server

Note, that they limit only reading, since usually you will not write
more data then you read, but sometimes it is possible in case of ALTER
UPDATE.

But for now, to keep things simple, I decided to limit this with only
2 settings instead of 4.

Note, that if the write throttling will be needed, then they can use the
same settings, and just create new throttler for write.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-12-24 22:31:49 +01:00
Azat Khuzhin
6ed9b53d1f Refactor test_throttling slightly for upcoming tests
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-12-24 22:29:43 +01:00
Azat Khuzhin
1168b7910b Update test_throttling with enabled s3_disable_checksum
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-11-26 20:03:33 +01:00
Azat Khuzhin
9c8096455e tests: switch test_throttling to S3 over https to make it more production like
One interesting thing about S3 C++ SDK is that it can read file multiple
times for calculating checksums and signature, but the last is not done
for the https protocol, though the checksum "could".

And indeed it does, since default checksum algorithm (MD5) does not
support streaming, and so it always calculated, regardless of the
protocol, however everything else (CRC*/SHA*) supports streamming and
actually will not be calculated for https at all!

This will be fixed in the follow up patch.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-10-08 07:31:33 +02:00
Azat Khuzhin
9fe4f1a934 Extensive coverage for bandwidth limiting settings
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-05-17 12:12:39 +02:00