János Benjamin Antal
d18b6c63d4
Change name of default azurite container to avoid clashing with azure blob storage tests
2024-08-15 09:41:03 +00:00
János Benjamin Antal
7b1bca2b48
Add missing assertion
2024-08-14 16:55:59 +00:00
János Benjamin Antal
61b96ed749
Make rest of the tests repeatable
2024-08-14 16:43:26 +00:00
János Benjamin Antal
4827b8bb1c
Make S3Queue tests repeatable
2024-08-14 14:56:02 +00:00
János Benjamin Antal
fc9929dc3d
Make tests with azurite repeatable
2024-08-14 14:15:05 +00:00
János Benjamin Antal
897b8d5a88
Try to give more chances to node2 to steal some work
2024-08-12 15:21:01 +00:00
Julia Kartseva
d6ecabb41d
Fix flaky test_storage_s3_queue/test.py::test_multiple_tables_streaming_sync_distributed
...
Disable parallel processing for the Ordered mode for the
test_storage_s3_queue/test.py::test_multiple_tables_streaming_sync_distributed
test.
The reason for this is that the load between the processing nodes is
too uneven when s3queue_processing_threads_num != 1,
e.g.:
```
$ grep res1 pytest.log
2024-08-07 07:15:58 [ 575 ] DEBUG : res1 size: 13300, res2 size: 1700, total_rows: 15000 (test.py:813, test_multiple_tables_streaming_sync_distributed)
```
In CIs environment, there are rare cases when one of the processors handles all the workload,
while the other is busy-waiting, and the test fails on assert:
When s3queue_processing_threads_num == 1, the workload is evenly
distributed:
```
$ grep res1 pytest.log
2024-08-07 07:26:52 [ 586 ] DEBUG : res1 size: 7200, res2 size: 7800, total_rows: 15000 (test.py:813, test_multiple_tables_streaming_sync_distributed)
```
This change only fixes test flakiness. Further investigation of the Order
mode parallelism is required.
2024-08-08 06:13:36 +00:00
János Benjamin Antal
0da44ad047
Merge pull request #67394 from ClickHouse/add-debug-to-s3-queue-shards-distributed
...
Print debug info in `test_storage_s3_queue/test.py::test_shards_distributed`
2024-08-05 12:23:40 +00:00
Pablo Marcos
15e0033016
Bring back the strict check
...
Also update the doc.
2024-07-31 12:09:52 +00:00
robot-clickhouse
f7f2a9872d
Automatic style fix
2024-07-29 16:02:14 +00:00
János Benjamin Antal
28ae671875
Print debug info
2024-07-29 15:52:25 +00:00
Pablo Marcos
4cf7e6a79d
Remove logging traces to help debugging
2024-07-29 11:15:03 +00:00
Pablo Marcos
5b9427dc56
Fix test flakiness
...
- Increase from 10s to 20s the max_age. Removes flakiness due to
the files in minio still present in following execution, even
if the cleanup stage of the test is clearly removing them.
- Use a busy-loop to check conditions directly, removing the
unnecessary sleep.
2024-07-29 10:36:18 +00:00
Pablo Marcos
59f9c12504
Increase sleep time make sure there is a new failure
...
The previous sleep was already adding +5s to make sure
the TTL was properly applied, so we'd rather use the
same value here instead of just 1s.
2024-07-25 14:37:50 +00:00
Pablo Marcos
f03d4bb7d5
Format with black
2024-07-24 15:10:25 +00:00
Pablo Marcos
91b7001df6
Refactor test to improve it
...
- Create wait_for_condition that checks greedily for a period of time
- Remove redundant checks
- Allow other tests running in parallel to have `ObjectStorageQueueFailedFiles` errors
2024-07-24 12:08:31 +00:00
Pablo Marcos
b25cad23ed
Use unique names for tables and files
2024-07-24 11:42:51 +00:00
kssenii
21f0eb2eec
Do not use async initialization of readers in s3queue
2024-07-02 18:13:49 +02:00
Kseniia Sumarokova
258e2f0421
Update test.py
2024-06-26 16:45:02 +02:00
kssenii
2ba697dcae
Merge remote-tracking branch 'origin' into add-azure-queue-storage
2024-06-26 13:05:56 +02:00
kssenii
95b1651ba6
Better pytest
2024-06-25 13:56:01 +02:00
kssenii
a066b78c4d
Merge remote-tracking branch 'origin' into s3queue-improvements
2024-06-21 12:27:32 +02:00
kssenii
5447145c7a
Fix test
2024-06-21 11:53:01 +02:00
robot-clickhouse
da55a213ef
Automatic style fix
2024-06-20 16:38:11 +00:00
kssenii
5904847316
Fix tests
2024-06-20 18:32:00 +02:00
robot-clickhouse
dc6fa85bc3
Automatic style fix
2024-06-19 14:37:55 +00:00
kssenii
c47c8d603e
Add AzureQueue
2024-06-19 16:30:57 +02:00
kssenii
78ccd03dd6
A few code renames in preparation for AzureQueue
2024-06-19 14:50:43 +02:00
Kseniia Sumarokova
959467f8fa
Update test.py
2024-06-19 14:03:24 +02:00
kssenii
1c415479f0
Do not release buckets until the files are commited
2024-06-18 18:58:38 +02:00
kssenii
0f2c2cc9bf
Fix tests
2024-06-17 17:11:17 +02:00
kssenii
366e801cea
Commit by limit
2024-06-14 14:24:06 +02:00
kssenii
1e435eb353
Fix tests
2024-06-11 19:01:24 +02:00
kssenii
bc294ef51f
S3Queue small fixes
2024-06-10 19:39:37 +02:00
kssenii
2a69c00d09
Merge remote-tracking branch 'origin/s3queue-fix-ordered-mode' into s3queue-fix-ordered-mode
2024-05-30 16:34:00 +02:00
kssenii
9238045eea
Add bucket versioning
2024-05-30 16:33:40 +02:00
robot-clickhouse
75bfa1746b
Automatic style fix
2024-05-30 13:07:17 +00:00
kssenii
b3a58a6294
Merge remote-tracking branch 'origin/s3queue-fix-ordered-mode' into s3queue-fix-ordered-mode
2024-05-30 14:58:12 +02:00
kssenii
ee4d5e4260
Fix remaining tests
2024-05-30 14:57:50 +02:00
robot-clickhouse
39dd2735a3
Automatic style fix
2024-05-29 19:44:48 +00:00
kssenii
59eed8086c
Fixes
2024-05-29 21:37:42 +02:00
robot-clickhouse
92fd1f08c9
Automatic style fix
2024-05-24 17:23:56 +00:00
kssenii
cfbf1cc1e2
S3Queue rework ordered mode
2024-05-24 11:47:32 +02:00
robot-clickhouse
98360e8c57
Automatic style fix
2024-05-14 14:37:26 +00:00
kssenii
cdf619b21e
Review fixes
2024-05-14 16:30:31 +02:00
Kseniia Sumarokova
1c527e86aa
Update test.py
2024-05-13 19:16:05 +02:00
kssenii
4b270e9078
Add a test
2024-05-13 17:15:26 +02:00
Dmitry Novik
b74a0359e5
Update integration tests
2024-03-20 13:43:18 +01:00
kssenii
d008ee725f
Add a test
2024-02-12 20:23:21 +01:00
robot-clickhouse
fe829cac48
Automatic style fix
2024-02-01 11:43:55 +00:00
