Commit Graph

12 Commits

Author SHA1 Message Date
Azat Khuzhin
837f4ea676 Add ability to throttle merges/mutations
Main motivation was to has an ability to throttle background tasks, to
avoid affecting queries.

To new server settings had been added for this:
- max_mutations_bandwidth_for_server
- max_merges_bandwidth_for_server

Note, that they limit only reading, since usually you will not write
more data then you read, but sometimes it is possible in case of ALTER
UPDATE.

But for now, to keep things simple, I decided to limit this with only
2 settings instead of 4.

Note, that if the write throttling will be needed, then they can use the
same settings, and just create new throttler for write.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-12-24 22:31:49 +01:00
Azat Khuzhin
6ed9b53d1f Refactor test_throttling slightly for upcoming tests
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-12-24 22:29:43 +01:00
Azat Khuzhin
1168b7910b Update test_throttling with enabled s3_disable_checksum
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-11-26 20:03:33 +01:00
Azat Khuzhin
2a27a8b0bf tests/test_throttling: check the maximum execution time as well
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-10-08 07:31:33 +02:00
Azat Khuzhin
c883da2b26 tests/test_throttling: adjust delays due to HTTPS usage
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-10-08 07:31:33 +02:00
Azat Khuzhin
9c8096455e tests: switch test_throttling to S3 over https to make it more production like
One interesting thing about S3 C++ SDK is that it can read file multiple
times for calculating checksums and signature, but the last is not done
for the https protocol, though the checksum "could".

And indeed it does, since default checksum algorithm (MD5) does not
support streaming, and so it always calculated, regardless of the
protocol, however everything else (CRC*/SHA*) supports streamming and
actually will not be calculated for https at all!

This will be fixed in the follow up patch.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-10-08 07:31:33 +02:00
serxa
135790c0d6 more fixes 2023-08-01 09:44:08 +00:00
serxa
513d3fd3c1 fix more places 2023-08-01 09:15:35 +00:00
serxa
fb8502ba76 do not test upper bounds for throttlers 2023-07-31 12:20:27 +00:00
Antonio Andelic
5da6c99f6d
Add comment 2023-07-24 12:02:27 +02:00
Antonio Andelic
3710c7238d Fix test_throttling 2023-07-24 09:19:06 +00:00
Azat Khuzhin
9fe4f1a934 Extensive coverage for bandwidth limiting settings
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-05-17 12:12:39 +02:00