kssenii
b7fa520781
Add s3queue check if cloud mode
2024-10-16 16:31:51 +02:00
Kseniia Sumarokova
752a1ec466
Merge pull request #70141 from ClickHouse/refactor-object-storage-read
...
Refactor reading from object storage
2024-10-03 10:14:28 +00:00
kssenii
0c00272d3f
Fix tests
2024-10-02 13:34:25 +02:00
robot-clickhouse
72d1c8e36d
Automatic style fix
2024-10-01 11:08:39 +00:00
kssenii
94c434086d
Fix
2024-10-01 13:02:31 +02:00
Kseniia Sumarokova
123d0cfa0b
Update test.py
2024-10-01 12:55:03 +02:00
Kseniia Sumarokova
719258a612
Update test.py
2024-10-01 12:48:04 +02:00
robot-clickhouse
7c6ce8611c
Automatic style fix
2024-09-27 10:19:49 +00:00
kssenii
f0a49e4b64
Merge remote-tracking branch 'origin' into fix-s3-queue-with-replicated-database
2024-09-26 12:44:20 +02:00
Kseniia Sumarokova
3a9c22bec0
Update test.py
2024-09-25 11:58:23 +02:00
kssenii
ebfc1a9b5a
Allow tables to start on upgrade 24.5 -> 24.6 if sharded mode was used
2024-09-19 13:58:59 +02:00
kssenii
54c2a8d0a7
Fix test
2024-09-19 12:38:08 +02:00
kssenii
a2450db286
Fix test
2024-09-18 20:03:45 +02:00
kssenii
373927d6a5
Fix tracked_files_ttl_sec
2024-09-18 19:25:18 +02:00
robot-clickhouse
ac0845a0a5
Automatic style fix
2024-09-17 10:54:56 +00:00
kssenii
f5861578d0
Update test
2024-09-17 12:48:51 +02:00
kssenii
c31ff58526
Adjust changes after merge with master
2024-09-16 13:01:28 +02:00
robot-clickhouse
c0a42ef877
Automatic style fix
2024-09-11 15:42:12 +00:00
Kseniia Sumarokova
abe8f29bf8
Update test
2024-09-11 17:32:20 +02:00
kssenii
3644adb589
Fix style check
2024-09-10 14:07:20 +02:00
kssenii
05ec50623c
Update test check
2024-09-10 13:55:33 +02:00
kssenii
e1ad411cb6
Fix
2024-09-09 13:42:38 +02:00
János Benjamin Antal
d18b6c63d4
Change name of default azurite container to avoid clashing with azure blob storage tests
2024-08-15 09:41:03 +00:00
János Benjamin Antal
7b1bca2b48
Add missing assertion
2024-08-14 16:55:59 +00:00
János Benjamin Antal
61b96ed749
Make rest of the tests repeatable
2024-08-14 16:43:26 +00:00
János Benjamin Antal
4827b8bb1c
Make S3Queue tests repeatable
2024-08-14 14:56:02 +00:00
János Benjamin Antal
fc9929dc3d
Make tests with azurite repeatable
2024-08-14 14:15:05 +00:00
János Benjamin Antal
897b8d5a88
Try to give more chances to node2 to steal some work
2024-08-12 15:21:01 +00:00
Julia Kartseva
d6ecabb41d
Fix flaky test_storage_s3_queue/test.py::test_multiple_tables_streaming_sync_distributed
...
Disable parallel processing for the Ordered mode for the
test_storage_s3_queue/test.py::test_multiple_tables_streaming_sync_distributed
test.
The reason for this is that the load between the processing nodes is
too uneven when s3queue_processing_threads_num != 1,
e.g.:
```
$ grep res1 pytest.log
2024-08-07 07:15:58 [ 575 ] DEBUG : res1 size: 13300, res2 size: 1700, total_rows: 15000 (test.py:813, test_multiple_tables_streaming_sync_distributed)
```
In CIs environment, there are rare cases when one of the processors handles all the workload,
while the other is busy-waiting, and the test fails on assert:
When s3queue_processing_threads_num == 1, the workload is evenly
distributed:
```
$ grep res1 pytest.log
2024-08-07 07:26:52 [ 586 ] DEBUG : res1 size: 7200, res2 size: 7800, total_rows: 15000 (test.py:813, test_multiple_tables_streaming_sync_distributed)
```
This change only fixes test flakiness. Further investigation of the Order
mode parallelism is required.
2024-08-08 06:13:36 +00:00
János Benjamin Antal
0da44ad047
Merge pull request #67394 from ClickHouse/add-debug-to-s3-queue-shards-distributed
...
Print debug info in `test_storage_s3_queue/test.py::test_shards_distributed`
2024-08-05 12:23:40 +00:00
Pablo Marcos
15e0033016
Bring back the strict check
...
Also update the doc.
2024-07-31 12:09:52 +00:00
robot-clickhouse
f7f2a9872d
Automatic style fix
2024-07-29 16:02:14 +00:00
János Benjamin Antal
28ae671875
Print debug info
2024-07-29 15:52:25 +00:00
Pablo Marcos
4cf7e6a79d
Remove logging traces to help debugging
2024-07-29 11:15:03 +00:00
Pablo Marcos
5b9427dc56
Fix test flakiness
...
- Increase from 10s to 20s the max_age. Removes flakiness due to
the files in minio still present in following execution, even
if the cleanup stage of the test is clearly removing them.
- Use a busy-loop to check conditions directly, removing the
unnecessary sleep.
2024-07-29 10:36:18 +00:00
Pablo Marcos
59f9c12504
Increase sleep time make sure there is a new failure
...
The previous sleep was already adding +5s to make sure
the TTL was properly applied, so we'd rather use the
same value here instead of just 1s.
2024-07-25 14:37:50 +00:00
Pablo Marcos
f03d4bb7d5
Format with black
2024-07-24 15:10:25 +00:00
Pablo Marcos
91b7001df6
Refactor test to improve it
...
- Create wait_for_condition that checks greedily for a period of time
- Remove redundant checks
- Allow other tests running in parallel to have `ObjectStorageQueueFailedFiles` errors
2024-07-24 12:08:31 +00:00
Pablo Marcos
b25cad23ed
Use unique names for tables and files
2024-07-24 11:42:51 +00:00
kssenii
21f0eb2eec
Do not use async initialization of readers in s3queue
2024-07-02 18:13:49 +02:00
Kseniia Sumarokova
258e2f0421
Update test.py
2024-06-26 16:45:02 +02:00
kssenii
2ba697dcae
Merge remote-tracking branch 'origin' into add-azure-queue-storage
2024-06-26 13:05:56 +02:00
kssenii
95b1651ba6
Better pytest
2024-06-25 13:56:01 +02:00
kssenii
a066b78c4d
Merge remote-tracking branch 'origin' into s3queue-improvements
2024-06-21 12:27:32 +02:00
kssenii
5447145c7a
Fix test
2024-06-21 11:53:01 +02:00
robot-clickhouse
da55a213ef
Automatic style fix
2024-06-20 16:38:11 +00:00
kssenii
5904847316
Fix tests
2024-06-20 18:32:00 +02:00
robot-clickhouse
dc6fa85bc3
Automatic style fix
2024-06-19 14:37:55 +00:00
kssenii
c47c8d603e
Add AzureQueue
2024-06-19 16:30:57 +02:00
kssenii
78ccd03dd6
A few code renames in preparation for AzureQueue
2024-06-19 14:50:43 +02:00
