Commit Graph

94 Commits

Author SHA1 Message Date
Julia Kartseva
d6ecabb41d Fix flaky test_storage_s3_queue/test.py::test_multiple_tables_streaming_sync_distributed
Disable parallel processing for the Ordered mode for the
test_storage_s3_queue/test.py::test_multiple_tables_streaming_sync_distributed
test.

The reason for this is that the load between the processing nodes is
too uneven when s3queue_processing_threads_num != 1,
e.g.:

```
$ grep res1 pytest.log
2024-08-07 07:15:58 [ 575 ] DEBUG : res1 size: 13300, res2 size: 1700, total_rows: 15000 (test.py:813, test_multiple_tables_streaming_sync_distributed)
```

In CIs environment, there are rare cases when one of the processors handles all the workload,
while the other is busy-waiting, and the test fails on assert:

When s3queue_processing_threads_num == 1, the workload is evenly
distributed:

```
$ grep res1 pytest.log
2024-08-07 07:26:52 [ 586 ] DEBUG : res1 size: 7200, res2 size: 7800, total_rows: 15000 (test.py:813, test_multiple_tables_streaming_sync_distributed)
```

This change only fixes test flakiness. Further investigation of the Order
mode parallelism is required.
2024-08-08 06:13:36 +00:00
János Benjamin Antal
0da44ad047
Merge pull request #67394 from ClickHouse/add-debug-to-s3-queue-shards-distributed
Print debug info in `test_storage_s3_queue/test.py::test_shards_distributed`
2024-08-05 12:23:40 +00:00
Pablo Marcos
15e0033016 Bring back the strict check
Also update the doc.
2024-07-31 12:09:52 +00:00
robot-clickhouse
f7f2a9872d Automatic style fix 2024-07-29 16:02:14 +00:00
János Benjamin Antal
28ae671875 Print debug info 2024-07-29 15:52:25 +00:00
Pablo Marcos
4cf7e6a79d Remove logging traces to help debugging 2024-07-29 11:15:03 +00:00
Pablo Marcos
5b9427dc56 Fix test flakiness
- Increase from 10s to 20s the max_age. Removes flakiness due to
  the files in minio still present in following execution, even
  if the cleanup stage of the test is clearly removing them.
- Use a busy-loop to check conditions directly, removing the
  unnecessary sleep.
2024-07-29 10:36:18 +00:00
Pablo Marcos
59f9c12504 Increase sleep time make sure there is a new failure
The previous sleep was already adding +5s to make sure
the TTL was properly applied, so we'd rather use the
same value here instead of just 1s.
2024-07-25 14:37:50 +00:00
Pablo Marcos
f03d4bb7d5 Format with black 2024-07-24 15:10:25 +00:00
Pablo Marcos
91b7001df6 Refactor test to improve it
- Create wait_for_condition that checks greedily for a period of time
- Remove redundant checks
- Allow other tests running in parallel to have `ObjectStorageQueueFailedFiles` errors
2024-07-24 12:08:31 +00:00
Pablo Marcos
b25cad23ed Use unique names for tables and files 2024-07-24 11:42:51 +00:00
kssenii
21f0eb2eec Do not use async initialization of readers in s3queue 2024-07-02 18:13:49 +02:00
Kseniia Sumarokova
258e2f0421
Update test.py 2024-06-26 16:45:02 +02:00
kssenii
2ba697dcae Merge remote-tracking branch 'origin' into add-azure-queue-storage 2024-06-26 13:05:56 +02:00
kssenii
95b1651ba6 Better pytest 2024-06-25 13:56:01 +02:00
kssenii
a066b78c4d Merge remote-tracking branch 'origin' into s3queue-improvements 2024-06-21 12:27:32 +02:00
kssenii
5447145c7a Fix test 2024-06-21 11:53:01 +02:00
robot-clickhouse
da55a213ef Automatic style fix 2024-06-20 16:38:11 +00:00
kssenii
5904847316 Fix tests 2024-06-20 18:32:00 +02:00
robot-clickhouse
dc6fa85bc3 Automatic style fix 2024-06-19 14:37:55 +00:00
kssenii
c47c8d603e Add AzureQueue 2024-06-19 16:30:57 +02:00
kssenii
78ccd03dd6 A few code renames in preparation for AzureQueue 2024-06-19 14:50:43 +02:00
Kseniia Sumarokova
959467f8fa
Update test.py 2024-06-19 14:03:24 +02:00
kssenii
1c415479f0 Do not release buckets until the files are commited 2024-06-18 18:58:38 +02:00
kssenii
0f2c2cc9bf Fix tests 2024-06-17 17:11:17 +02:00
kssenii
366e801cea Commit by limit 2024-06-14 14:24:06 +02:00
kssenii
1e435eb353 Fix tests 2024-06-11 19:01:24 +02:00
kssenii
bc294ef51f S3Queue small fixes 2024-06-10 19:39:37 +02:00
kssenii
2a69c00d09 Merge remote-tracking branch 'origin/s3queue-fix-ordered-mode' into s3queue-fix-ordered-mode 2024-05-30 16:34:00 +02:00
kssenii
9238045eea Add bucket versioning 2024-05-30 16:33:40 +02:00
robot-clickhouse
75bfa1746b Automatic style fix 2024-05-30 13:07:17 +00:00
kssenii
b3a58a6294 Merge remote-tracking branch 'origin/s3queue-fix-ordered-mode' into s3queue-fix-ordered-mode 2024-05-30 14:58:12 +02:00
kssenii
ee4d5e4260 Fix remaining tests 2024-05-30 14:57:50 +02:00
robot-clickhouse
39dd2735a3 Automatic style fix 2024-05-29 19:44:48 +00:00
kssenii
59eed8086c Fixes 2024-05-29 21:37:42 +02:00
robot-clickhouse
92fd1f08c9 Automatic style fix 2024-05-24 17:23:56 +00:00
kssenii
cfbf1cc1e2 S3Queue rework ordered mode 2024-05-24 11:47:32 +02:00
robot-clickhouse
98360e8c57 Automatic style fix 2024-05-14 14:37:26 +00:00
kssenii
cdf619b21e Review fixes 2024-05-14 16:30:31 +02:00
Kseniia Sumarokova
1c527e86aa
Update test.py 2024-05-13 19:16:05 +02:00
kssenii
4b270e9078 Add a test 2024-05-13 17:15:26 +02:00
Dmitry Novik
b74a0359e5 Update integration tests 2024-03-20 13:43:18 +01:00
kssenii
12cd566681 Update config for test 2024-03-19 11:47:53 +01:00
kssenii
d008ee725f Add a test 2024-02-12 20:23:21 +01:00
robot-clickhouse
fe829cac48 Automatic style fix 2024-02-01 11:43:55 +00:00
kssenii
3a16427e00 Fix test 2024-02-01 12:35:01 +01:00
kssenii
f7a5f09a0f Add one more test 2024-01-31 22:59:25 +01:00
kssenii
361b2f107b Allow to define a starting point for s3queue ordered mode 2024-01-31 21:28:18 +01:00
kssenii
145f6a31cb Fix 2024-01-30 16:29:40 +01:00
kssenii
df7d3b8a40 Allocate current shard number from keeper 2024-01-29 19:36:16 +01:00