thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-15 19:02:04 +00:00
Code Issues Packages Projects Releases Wiki Activity
160,417 Commits 9,326 Branches 1,783 Tags 2.1 GiB
dc7bc42223
Commit Graph

165 Commits

Author SHA1 Message Date
robot-clickhouse
7c6ce8611c Automatic style fix 2024-09-27 10:19:49 +00:00
kssenii
f0a49e4b64 Merge remote-tracking branch 'origin' into fix-s3-queue-with-replicated-database 2024-09-26 12:44:20 +02:00
Kseniia Sumarokova
3a9c22bec0
Update test.py 2024-09-25 11:58:23 +02:00
kssenii
ebfc1a9b5a Allow tables to start on upgrade 24.5 -> 24.6 if sharded mode was used 2024-09-19 13:58:59 +02:00
kssenii
54c2a8d0a7 Fix test 2024-09-19 12:38:08 +02:00
kssenii
a2450db286 Fix test 2024-09-18 20:03:45 +02:00
kssenii
373927d6a5 Fix tracked_files_ttl_sec 2024-09-18 19:25:18 +02:00
robot-clickhouse
ac0845a0a5 Automatic style fix 2024-09-17 10:54:56 +00:00
kssenii
f5861578d0 Update test 2024-09-17 12:48:51 +02:00
kssenii
c31ff58526 Adjust changes after merge with master 2024-09-16 13:01:28 +02:00
robot-clickhouse
c0a42ef877 Automatic style fix 2024-09-11 15:42:12 +00:00
Kseniia Sumarokova
abe8f29bf8
Update test 2024-09-11 17:32:20 +02:00
kssenii
3644adb589 Fix style check 2024-09-10 14:07:20 +02:00
kssenii
05ec50623c Update test check 2024-09-10 13:55:33 +02:00
kssenii
e1ad411cb6 Fix 2024-09-09 13:42:38 +02:00
János Benjamin Antal
d18b6c63d4 Change name of default azurite container to avoid clashing with azure blob storage tests 2024-08-15 09:41:03 +00:00
János Benjamin Antal
7b1bca2b48 Add missing assertion 2024-08-14 16:55:59 +00:00
János Benjamin Antal
61b96ed749 Make rest of the tests repeatable 2024-08-14 16:43:26 +00:00
János Benjamin Antal
4827b8bb1c Make S3Queue tests repeatable 2024-08-14 14:56:02 +00:00
János Benjamin Antal
fc9929dc3d Make tests with azurite repeatable 2024-08-14 14:15:05 +00:00
János Benjamin Antal
897b8d5a88 Try to give more chances to node2 to steal some work 2024-08-12 15:21:01 +00:00
Julia Kartseva
d6ecabb41d Fix flaky test_storage_s3_queue/test.py::test_multiple_tables_streaming_sync_distributed 
Disable parallel processing for the Ordered mode for the
test_storage_s3_queue/test.py::test_multiple_tables_streaming_sync_distributed
test.

The reason for this is that the load between the processing nodes is
too uneven when s3queue_processing_threads_num != 1,
e.g.:

```
$ grep res1 pytest.log
2024-08-07 07:15:58 [ 575 ] DEBUG : res1 size: 13300, res2 size: 1700, total_rows: 15000 (test.py:813, test_multiple_tables_streaming_sync_distributed)
```

In CIs environment, there are rare cases when one of the processors handles all the workload,
while the other is busy-waiting, and the test fails on assert:

When s3queue_processing_threads_num == 1, the workload is evenly
distributed:

```
$ grep res1 pytest.log
2024-08-07 07:26:52 [ 586 ] DEBUG : res1 size: 7200, res2 size: 7800, total_rows: 15000 (test.py:813, test_multiple_tables_streaming_sync_distributed)
```

This change only fixes test flakiness. Further investigation of the Order
mode parallelism is required.
 2024-08-08 06:13:36 +00:00
János Benjamin Antal
0da44ad047
Merge pull request #67394 from ClickHouse/add-debug-to-s3-queue-shards-distributed 
Print debug info in `test_storage_s3_queue/test.py::test_shards_distributed`
 2024-08-05 12:23:40 +00:00
Pablo Marcos
15e0033016 Bring back the strict check 
Also update the doc.
 2024-07-31 12:09:52 +00:00
robot-clickhouse
f7f2a9872d Automatic style fix 2024-07-29 16:02:14 +00:00
János Benjamin Antal
28ae671875 Print debug info 2024-07-29 15:52:25 +00:00
Pablo Marcos
4cf7e6a79d Remove logging traces to help debugging 2024-07-29 11:15:03 +00:00
Pablo Marcos
5b9427dc56 Fix test flakiness 
- Increase from 10s to 20s the max_age. Removes flakiness due to
  the files in minio still present in following execution, even
  if the cleanup stage of the test is clearly removing them.
- Use a busy-loop to check conditions directly, removing the
  unnecessary sleep.
 2024-07-29 10:36:18 +00:00
Pablo Marcos
59f9c12504 Increase sleep time make sure there is a new failure 
The previous sleep was already adding +5s to make sure
the TTL was properly applied, so we'd rather use the
same value here instead of just 1s.
 2024-07-25 14:37:50 +00:00
Pablo Marcos
f03d4bb7d5 Format with black 2024-07-24 15:10:25 +00:00
Pablo Marcos
91b7001df6 Refactor test to improve it 
- Create wait_for_condition that checks greedily for a period of time
- Remove redundant checks
- Allow other tests running in parallel to have `ObjectStorageQueueFailedFiles` errors
 2024-07-24 12:08:31 +00:00
Pablo Marcos
b25cad23ed Use unique names for tables and files 2024-07-24 11:42:51 +00:00
kssenii
21f0eb2eec Do not use async initialization of readers in s3queue 2024-07-02 18:13:49 +02:00
Kseniia Sumarokova
258e2f0421
Update test.py 2024-06-26 16:45:02 +02:00
kssenii
2ba697dcae Merge remote-tracking branch 'origin' into add-azure-queue-storage 2024-06-26 13:05:56 +02:00
kssenii
95b1651ba6 Better pytest 2024-06-25 13:56:01 +02:00
kssenii
a066b78c4d Merge remote-tracking branch 'origin' into s3queue-improvements 2024-06-21 12:27:32 +02:00
kssenii
5447145c7a Fix test 2024-06-21 11:53:01 +02:00
robot-clickhouse
da55a213ef Automatic style fix 2024-06-20 16:38:11 +00:00
kssenii
5904847316 Fix tests 2024-06-20 18:32:00 +02:00
robot-clickhouse
dc6fa85bc3 Automatic style fix 2024-06-19 14:37:55 +00:00
kssenii
c47c8d603e Add AzureQueue 2024-06-19 16:30:57 +02:00
kssenii
78ccd03dd6 A few code renames in preparation for AzureQueue 2024-06-19 14:50:43 +02:00
Kseniia Sumarokova
959467f8fa
Update test.py 2024-06-19 14:03:24 +02:00
kssenii
1c415479f0 Do not release buckets until the files are commited 2024-06-18 18:58:38 +02:00
kssenii
0f2c2cc9bf Fix tests 2024-06-17 17:11:17 +02:00
kssenii
366e801cea Commit by limit 2024-06-14 14:24:06 +02:00
kssenii
1e435eb353 Fix tests 2024-06-11 19:01:24 +02:00
kssenii
bc294ef51f S3Queue small fixes 2024-06-10 19:39:37 +02:00
kssenii
2a69c00d09 Merge remote-tracking branch 'origin/s3queue-fix-ordered-mode' into s3queue-fix-ordered-mode 2024-05-30 16:34:00 +02:00
kssenii
9238045eea Add bucket versioning 2024-05-30 16:33:40 +02:00
robot-clickhouse
75bfa1746b Automatic style fix 2024-05-30 13:07:17 +00:00
kssenii
b3a58a6294 Merge remote-tracking branch 'origin/s3queue-fix-ordered-mode' into s3queue-fix-ordered-mode 2024-05-30 14:58:12 +02:00
kssenii
ee4d5e4260 Fix remaining tests 2024-05-30 14:57:50 +02:00
robot-clickhouse
39dd2735a3 Automatic style fix 2024-05-29 19:44:48 +00:00
kssenii
59eed8086c Fixes 2024-05-29 21:37:42 +02:00
robot-clickhouse
92fd1f08c9 Automatic style fix 2024-05-24 17:23:56 +00:00
kssenii
cfbf1cc1e2 S3Queue rework ordered mode 2024-05-24 11:47:32 +02:00
robot-clickhouse
98360e8c57 Automatic style fix 2024-05-14 14:37:26 +00:00
kssenii
cdf619b21e Review fixes 2024-05-14 16:30:31 +02:00
Kseniia Sumarokova
1c527e86aa
Update test.py 2024-05-13 19:16:05 +02:00
kssenii
4b270e9078 Add a test 2024-05-13 17:15:26 +02:00
Dmitry Novik
b74a0359e5 Update integration tests 2024-03-20 13:43:18 +01:00
kssenii
12cd566681 Update config for test 2024-03-19 11:47:53 +01:00
kssenii
d008ee725f Add a test 2024-02-12 20:23:21 +01:00
robot-clickhouse
fe829cac48 Automatic style fix 2024-02-01 11:43:55 +00:00
kssenii
3a16427e00 Fix test 2024-02-01 12:35:01 +01:00
kssenii
f7a5f09a0f Add one more test 2024-01-31 22:59:25 +01:00
kssenii
361b2f107b Allow to define a starting point for s3queue ordered mode 2024-01-31 21:28:18 +01:00
kssenii
145f6a31cb Fix 2024-01-30 16:29:40 +01:00
kssenii
df7d3b8a40 Allocate current shard number from keeper 2024-01-29 19:36:16 +01:00
kssenii
ef252d8376 Validate settings with keeper, add a test 2024-01-29 12:36:00 +01:00
kssenii
7f8f379d7f Parallel & disrtibuted processing for ordered mode 2024-01-24 16:32:15 +01:00
robot-clickhouse
29173a602e Automatic style fix 2024-01-08 02:39:19 +00:00
Sema Checherinda
d5f86f671d fix and test that S3Clients reused 2024-01-07 02:19:06 +01:00
kssenii
529a1d67cb Fix 2023-12-28 12:48:56 +01:00
Alexey Milovidov
76bbeaa676 S3Queue is production ready 2023-12-05 21:10:19 +01:00
Kseniia Sumarokova
4c7daf5167
Fix 2023-11-20 14:55:23 +01:00
Kseniia Sumarokova
2a1467b8da
Update test_storage_s3_queue/test.py 2023-11-16 19:33:17 +01:00
Kseniia Sumarokova
e48df94772
Merge pull request #56377 from ClickHouse/minor-improvements-for-s3-queue 
Minor improvements for S3Queue
 2023-11-10 11:28:09 +01:00
Kseniia Sumarokova
b05a66b9b5
Update test.py 2023-11-09 13:04:46 +01:00
robot-clickhouse
c064488bbf Automatic style fix 2023-11-08 11:48:09 +00:00
kssenii
dfcc94fff0 Fix test 2023-11-08 12:29:40 +01:00
Kseniia Sumarokova
cc3144ea2d
Merge pull request #56370 from ClickHouse/kssenii-patch-6 
Update test_storage_s3_queue/test.py
 2023-11-07 16:29:34 +01:00
robot-clickhouse
d55a0bd686 Automatic style fix 2023-11-06 15:04:45 +00:00
kssenii
f2c0434c4d Merge remote-tracking branch 'origin/master' into minor-improvements-for-s3-queue 2023-11-06 15:51:11 +01:00
kssenii
8db2aca832 Add a test 2023-11-06 15:48:04 +01:00
kssenii
3ecce143fc Restrict modes for the test 2023-11-06 14:44:24 +01:00
Kseniia Sumarokova
036922e3b4
Update test.py 2023-11-06 12:21:13 +01:00
kssenii
5d700605a5 Merge remote-tracking branch 'upstream/master' into fix-flaky-s3-queue-test 2023-10-26 17:18:26 +02:00
helifu
4015176d27
A small fix (#55938) 2023-10-26 13:19:58 +02:00
kssenii
de8064da6e Debug info 2023-10-25 12:31:00 +02:00
kssenii
b6b124f5a9 Usability improvement 2023-10-13 14:16:56 +02:00
kssenii
c579f5b321 Fix style check 2023-09-27 20:10:05 +02:00
kssenii
e0ff76a7f9 Fix 2023-09-27 18:44:53 +02:00
kssenii
3b54b6be88 Add system tables 2023-09-25 20:04:15 +02:00
kssenii
e4256eb268 Better 2023-09-25 13:44:45 +02:00
kssenii
6846fe3c58 Fxi 2023-09-15 14:21:08 +02:00
kssenii
342755d35e Better 2023-09-14 18:41:31 +02:00
kssenii
220a67eca7 Better 2023-09-08 16:38:05 +02:00