thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-20 06:32:08 +00:00
Code Issues Packages Projects Releases Wiki Activity
93,074 Commits 9,144 Branches 1,778 Tags 2 GiB
ae16555aa4
Commit Graph

4524 Commits

Author SHA1 Message Date
lgbo-ustc
efec6f9ac5 support alter on storagehive 2022-06-20 14:57:33 +08:00
lgbo-ustc
c13bf03fe0 fixed code style 2022-06-20 09:56:21 +08:00
Vitaly Baranov
2c8788266c Fix tests. 2022-06-20 03:44:49 +02:00
Vitaly Baranov
8a7c970ce0 Fix style. 2022-06-19 15:58:26 +02:00
Vitaly Baranov
b0d19328b8 Add test for ReplicatedAccessStorage. 2022-06-19 14:14:41 +02:00
Vitaly Baranov
115be82440 DiskAccessStorage is now allowed to backup by default. 2022-06-19 13:16:36 +02:00
Vitaly Baranov
9f197defda Add support for setting 'allow_backup' to skip access entities from putting to backup. 2022-06-19 12:49:50 +02:00
Vitaly Baranov
01aaaf7395 More accurate access checking for RESTORE. 2022-06-19 11:26:41 +02:00
Vitaly Baranov
de9a07d18d Fix RESTORE ALL for tables without database in backup. 2022-06-18 14:07:01 +02:00
Vitaly Baranov
36475c5b98 Fix handling empty files in backups. 2022-06-18 12:28:32 +02:00
Vitaly Baranov
a0c558a17e Implement backup/restore for ACL system tables (system.users, system.roles, etc.) 2022-06-17 18:14:31 +02:00
mergify[bot]
23e0b16058
Merge branch 'master' into fix-possible-crash-after-removing-replica-in-distributed 2022-06-17 11:50:44 +00:00
mergify[bot]
296905771f
Merge branch 'master' into better-support-gcp 2022-06-17 01:19:11 +00:00
Azat Khuzhin
68b4f3fbb3 Add backward compatiblity test for INSERT w/ and w/o ProfileEvents 
v2: apply black formatting (sigh)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-16 11:59:01 +03:00
Azat Khuzhin
20f0602c20 tests/integration: add ability to query specific host 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-16 11:59:01 +03:00
lgbo-ustc
35d534c213 nested struct in struct 2022-06-16 16:45:05 +08:00
lgbo-ustc
4f13521aa6 struct type support for storage hive 2022-06-16 09:35:34 +08:00
mergify[bot]
7f24574609
Merge branch 'master' into better-support-gcp 2022-06-15 23:18:20 +00:00
Alexander Tokmakov
2297708032 use random container name 2022-06-15 23:08:42 +02:00
Vitaly Baranov
c2c35fad82 Refactoring of the code getting create table queries for backup. 2022-06-15 20:32:35 +02:00
Vitaly Baranov
1198e86295 Fix storing temporary tables and skipping system tables while making a backup. 2022-06-15 20:32:34 +02:00
Vitaly Baranov
b2323991d6 BACKUP DATABASE now stores inner tables of matviews only if the corresponding matviews are stored. 2022-06-15 20:32:34 +02:00
Vitaly Baranov
6877b8f864 Fix renaming visitor. 2022-06-15 20:32:34 +02:00
Vitaly Baranov
d78a2cda72 Restore tables regarding their dependencies. 2022-06-15 20:32:34 +02:00
Vitaly Baranov
ce1836f0d2 Lock tables for share before backup and restore. 2022-06-15 20:28:43 +02:00
Vitaly Baranov
131382e025 Fix async tests. 2022-06-15 20:28:42 +02:00
Anton Popov
eea7e4eced fix test 2022-06-15 15:02:02 +00:00
alesapin
af1cd745e1
Merge pull request #37975 from kssenii/clean-up-broken-detached 
Clean up broken detached parts after timeout
 2022-06-14 20:53:31 +02:00
alesapin
8d45d1e71b
Fix setting name in test 2022-06-14 11:52:44 +02:00
Antonio Andelic
dded528d13
Merge pull request #36424 from PolyProgrammist/r1unfreeze 
SYSTEM UNFREEZE query that deletes the whole backup
 2022-06-14 08:44:45 +02:00
kssenii
74cffaf186 Merge master 2022-06-14 00:45:19 +02:00
kssenii
66ec0e41fa Fix test 2022-06-14 00:17:00 +02:00
Vitaly Baranov
241b51c7d4
Add implicit grants with grant option too. (#38017) 2022-06-14 00:09:51 +02:00
kssenii
11c624914a Fix black 2022-06-13 21:46:57 +02:00
Nikolai Kochetov
43dff9d34c Black 2022-06-13 19:44:12 +00:00
kssenii
34e63b20fb Review fixes 2022-06-13 21:27:54 +02:00
Nikolai Kochetov
695d586604 Fix possible crash in Distributed async insert in case of removing a replica from config. 2022-06-13 15:21:43 +00:00
Antonio Andelic
b9cf6fe367 Merge branch 'master' into pp-system-unfreeze 2022-06-13 14:46:30 +00:00
kssenii
debe467ee7 Review fixes 2022-06-13 14:25:42 +02:00
kssenii
4859e11c27 Review fixes 2022-06-12 01:15:10 +02:00
Alexander Tokmakov
72fe5144ca
Revert "Don't try to kill empty list of containers in integration/runner" 2022-06-11 22:55:31 +03:00
Alexey Milovidov
384cef7242
Merge pull request #37854 from excitoon-favorites/integrationtestsfixes 
Don't try to kill empty list of containers in `integration/runner`
 2022-06-11 03:01:41 +03:00
Alexander Tokmakov
2319ef7647 fix test 2022-06-10 17:48:14 +02:00
kssenii
fd6c84b831 Review fixes 2022-06-10 14:23:13 +02:00
kssenii
498f389c21 Merge master 2022-06-10 12:49:10 +02:00
kssenii
7a2676c7ab Clean up broken detached parts with timeout 2022-06-10 12:27:57 +02:00
alesapin
36924cb47d
Merge pull request #37957 from ClickHouse/tavplubix-patch-1 
Fix flaky `test_parts_delete_zookeeper`
 2022-06-10 11:47:55 +02:00
Roman Vasin
4c560584c7 Code cleanup in KerberosInit and kafka integration tests 2022-06-10 12:38:39 +03:00
Alexander Tokmakov
780f7c87c7
Merge pull request #36113 from ClickHouse/remove-useless-code-2 
Remove useless code in ReplicatedMergeTreeRestartingThread
 2022-06-09 17:56:28 +03:00
Alexander Tokmakov
68a0950086
Update test.py 2022-06-09 17:53:52 +03:00
Maksim Kita
aa5293da5d
Merge pull request #37947 from kitaisreal/dictionaries-update-field-condition-fix 
Dictionaries custom query with update field fix
 2022-06-09 16:31:07 +02:00
Maksim Kita
a9d568c63c Dictionaries custom query with update field fix 2022-06-09 11:51:07 +02:00
Alexander Tokmakov
0186143792
Update test.py 2022-06-08 15:49:37 +03:00
Vladimir Chebotarev
7626854ef3 Typos. 2022-06-08 15:47:20 +03:00
Vladimir Chebotarev
eb9117804e Better. 2022-06-08 15:47:20 +03:00
Vladimir Chebotarev
7f5e9a2a90 Better. 2022-06-08 15:47:20 +03:00
Vladimir Chebotarev
6749ba14de Don't try to kill empty list of containers in integration/runner. 2022-06-08 15:47:20 +03:00
Vadim Volodin
637d293fbd Add SYSTEM UNFREEZE query 2022-06-08 15:21:14 +03:00
Alexander Tokmakov
627af2a56b
Update test.py 2022-06-08 14:29:35 +03:00
Alexander Tokmakov
e3cc4afdf7
Update test.py 2022-06-08 14:13:57 +03:00
alesapin
9fdc783eac
Merge pull request #37906 from ClickHouse/remove_tmp_directories_on_replicated 
Remove temporary `delete_tmp_` on clickhouse server start
 2022-06-08 12:35:51 +02:00
Roman Vasin
2b76d0c6a9 Add new integration test for kerberized Kafka; remove old kinit code from HDFSCommon 2022-06-08 12:26:35 +03:00
Kseniia Sumarokova
6211a1c390
Merge pull request #37801 from azat/s3-fix-empty-read 
Fix reading of empty S3 files
 2022-06-07 23:36:14 +02:00
alesapin
8b26953e3f
Merge pull request #37908 from ClickHouse/tavplubix-patch-1 
Try fix `test_grpc_protocol/test.py::test_progress`
 2022-06-07 21:48:00 +02:00
Azat Khuzhin
8ddf277670 tests: test_log_family_s3 apply black 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-07 20:41:26 +03:00
Azat Khuzhin
ee45bb3c65 tests: use id for parametrized tests in test_log_family_s3 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-07 20:41:26 +03:00
Azat Khuzhin
499ff70295 tests: improve test_log_family_s3 in case of failures 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-06-07 20:41:26 +03:00
alesapin
96b454e1b0 BLACK 2022-06-07 17:58:53 +02:00
alesapin
e32d36d790 Proper fix 2022-06-07 17:58:32 +02:00
Alexander Tokmakov
a1e69a287f
Update test.py 2022-06-07 18:57:26 +03:00
alesapin
5741ac2b9e Remove redundant line 2022-06-07 17:41:14 +02:00
alesapin
278e7722ed Review fix 2022-06-07 16:09:21 +02:00
Roman Vasin
a156a77890 Add KerberosInit into StorageKafka 2022-06-07 14:59:46 +03:00
alesapin
b459e33892 Merge branch 'master' into fix_keeper_converter_tests 2022-06-07 13:08:40 +02:00
Kseniia Sumarokova
4272ca8836
Merge pull request #36060 from Varinara/master 
Add feature disks (ls, C, list-disks, disk, help)
 2022-06-07 11:22:23 +02:00
Roman Vasin
cb53aa15ec Fix HDFSCommon and test_storage_kerberized_hdfs to make running integration tests 2022-06-07 12:06:22 +03:00
Vitaly Baranov
d199478169
Merge pull request #37303 from ClickHouse/fix_trash 
Try to fix some trash
 2022-06-07 10:17:39 +02:00
Robert Schulze
07ba122572
Merge pull request #37642 from amosbird/less-flaky-jbod-test 
Less flaky jbod rebalancer test
 2022-06-07 09:16:09 +02:00
alesapin
2329407d6f BLACK 2022-06-07 00:42:56 +02:00
alesapin
a4632bb6ae Fix keeper converter test 2022-06-06 22:59:34 +02:00
mergify[bot]
7a76a7a606
Merge branch 'master' into master 2022-06-06 19:44:32 +00:00
alesapin
5faaf1b553
Merge pull request #37884 from nickitat/update_protobuf_files 
Update protobuf files for kafka and rabbitmq [fix integration tests]
 2022-06-06 20:57:21 +02:00
Varinara
ed6e8176fe Add basic commands for disk tool (list-disks, list, move, remove, link, copy, read, write) + tests 2022-06-06 16:52:58 +03:00
Nickita Taranov
9fd9836237 update other files 2022-06-06 15:31:20 +02:00
Nickita Taranov
90f023d65f impl 2022-06-06 13:09:48 +02:00
Alexander Tokmakov
13c6cf9b95 fix style check 2022-06-03 18:16:42 +02:00
Maksim Kita
b4cc8ce22b Fixed style check 2022-06-03 18:06:59 +02:00
Maksim Kita
23b92cbe10 Fixed style check 2022-06-03 18:06:59 +02:00
Maksim Kita
c4da2540e9 Added integration tests 2022-06-03 18:06:59 +02:00
mergify[bot]
ddf7210ecc
Merge branch 'master' into remove-useless-code-2 2022-06-03 13:58:45 +00:00
Alexander Tokmakov
21271726de
Merge pull request #37740 from ClickHouse/try_fix_tests2 
Some fixes for tests
 2022-06-02 16:12:46 +03:00
Kseniia Sumarokova
2f777b7a4b
Merge pull request #37690 from kssenii/fix-mysql-db 
Fix segfault with mysql db + show create table + named collections
 2022-06-02 12:00:42 +02:00
Chebarykov Pavel
6353b58b6a Fix startup test 2022-06-02 09:09:24 +00:00
Maksim Kita
cb93135326
Merge pull request #37711 from kitaisreal/executable-user-defined-functions-nullable-arguments-fix 
Fix executable user default functions execution with Nullable arguments
 2022-06-02 11:05:34 +02:00
alesapin
b7e8bbb154
Merge pull request #37679 from ClickHouse/fix-keeper-recovery-test 
Fix `test_keeper_force_recovery*` tests
 2022-06-01 18:45:32 +02:00
Yakov Olkhovskiy
e23cec01d5
Merge pull request #37581 from ClickHouse/http-named-collection 
Support for HTTP source for Data Dictionaries in Named Collections
 2022-06-01 11:55:04 -04:00
Dmitry Novik
7fbe91ca81
Merge pull request #37460 from ClickHouse/memory-overcommit-improvement 
Memory Overcommit: update defaults, exception message and add ProfileEvent
 2022-06-01 17:06:33 +02:00
Amos Bird
139264d881
Less parallelism instead 2022-06-01 23:02:31 +08:00
Kseniia Sumarokova
503d94f73b
Update test.py 2022-06-01 14:44:46 +02:00
Alexander Tokmakov
75f49a48e1 Merge branch 'master' into fix_trash 2022-06-01 14:20:46 +02:00
Alexander Tokmakov
153862dc31 fix tests 
(cherry picked from commit ed52f463d95ac49431600f167a08507d7fcaaaf0)
 2022-06-01 13:47:52 +02:00
Maksim Kita
9aee3f3156 Fixed tests 2022-06-01 13:06:44 +02:00
Maksim Kita
66f43b9ad3 Fix executable user default functions execution with Nullable arguments 2022-05-31 18:46:33 +02:00
Yakov Olkhovskiy
4b427336e3 tests with overridden and appended parameters 2022-05-31 09:37:34 -04:00
mergify[bot]
ba49c6bb46
Merge branch 'master' into memory-overcommit-improvement 2022-05-31 13:17:06 +00:00
kssenii
c2087b3145 Fix 2022-05-31 14:38:11 +02:00
kssenii
69cd3a2b10 Fix 2022-05-31 14:20:31 +02:00
Antonio Andelic
582be42329 Wait for leader election 2022-05-31 11:53:46 +00:00
Alexander Tokmakov
30a7b07d97
Merge pull request #37658 from vitlibar/fix-flaky-test_row_policy 
Fix flaky test test_row_policy
 2022-05-31 12:59:50 +03:00
mergify[bot]
d4e722bbfa
Merge branch 'master' into http-named-collection 2022-05-30 16:40:18 +00:00
Vitaly Baranov
486a11a5e2 Fix flaky test test_row_policy. 2022-05-30 18:34:28 +02:00
Kruglov Pavel
0615866aea
Merge pull request #37450 from Avogar/check-format-on-storage-creation 
Check format name on storage creation
 2022-05-30 14:23:20 +02:00
Amos Bird
030fe6e272
Less flaky jbod rebalancer test 2022-05-30 16:53:55 +08:00
Alexander Tokmakov
4e52f45695 Merge branch 'master' into fix_trash 2022-05-28 19:43:19 +02:00
Yakov Olkhovskiy
4f07c684da style fix 2022-05-28 12:45:53 -04:00
Yakov Olkhovskiy
3ff32fe81d test is added 2022-05-28 12:30:05 -04:00
Dmitry Novik
db2fe33926
Update test.py 2022-05-27 20:43:59 +02:00
Dmitry Novik
60b9d81773 Remove global_memory_usage_overcommit_max_wait_microseconds 2022-05-27 16:30:29 +00:00
tchepavel
42cd854688
Merge branch 'master' into nats-integration 2022-05-27 16:05:08 +03:00
alesapin
461bb42fb8 Fix flaky test 2022-05-27 13:40:07 +02:00
tchepavel
531bda5a65 Spelling and more black python 2022-05-25 22:13:26 +03:00
tchepavel
d4afc36f8e Style+build fixes; make long tests time limited 2022-05-25 22:02:47 +03:00
alesapin
6f5c86e55e Merge branch 'master' into i_object_storage 2022-05-25 20:49:01 +02:00
tchepavel
0eae83aebc Merge branch 'nats-integration' of github.com:tchepavel/ClickHouse into nats-integration 2022-05-25 21:32:26 +03:00
tchepavel
681695b706 Black python 2022-05-25 21:31:14 +03:00
alesapin
0a3597da72
Merge pull request #34915 from ianton-ru/MDB-16962 
Fix collision of S3 operation log revision
 2022-05-25 18:15:31 +02:00
avogar
9518c41dda Try to fix tests 2022-05-25 09:01:12 +00:00
Alexander Tokmakov
bb5f04efaf Merge branch 'master' into fix_trash 2022-05-24 17:45:51 +02:00
tchepavel
43226a57fc
Merge branch 'master' into nats-integration 2022-05-24 18:15:11 +03:00
tchepavel
fc83f97199 Fix same names in proto test 2022-05-24 18:14:16 +03:00
Alexander Tokmakov
d0f998fb88 Merge branch 'master' into fix_trash 2022-05-23 21:25:56 +02:00
Antonio Andelic
e91e7fdba7 Fix style 2022-05-23 15:33:29 +00:00
tchepavel
d9436ec7dd
Merge branch 'master' into nats-integration 2022-05-23 14:30:09 +03:00
Antonio Andelic
c268296fc6 Fix single node force recovery and add tests 2022-05-23 09:53:46 +00:00
mergify[bot]
747aa5575c
Merge branch 'master' into remove-useless-code-2 2022-05-22 17:41:57 +00:00
alesapin
d414d85654 Fix hdfs bug 2022-05-20 17:43:48 +02:00
alesapin
654b27e307 Merge branch 'master' into i_object_storage 2022-05-20 11:56:13 +02:00
Sergei Trifonov
2450ab9079
Merge pull request #36686 from azat/disk-least-used 
Implement least_used load balancing algorithm for disks inside volume
 2022-05-20 09:16:07 +02:00
Robert Schulze
c0d5705891
Merge pull request #37326 from ClickHouse/fix-marks-cache-size-cfg 
Provide default value for mark_cache_size
 2022-05-19 20:20:25 +02:00
Kseniia Sumarokova
d4ad138a04
Merge pull request #37103 from bigo-sg/hive_partition_key_read 
optimization for reading hive file  when all columns to read are partition keys
 2022-05-19 14:24:00 +02:00
Kseniia Sumarokova
af2d9154e1
Merge pull request #37279 from bigo-sg/hive_local_cache_multi_disk 
Support multi disks for caching hive files
 2022-05-19 13:25:03 +02:00
Azat Khuzhin
ba26b3cf4c Implement lead_used load balancing algorithm for disks inside volume 
v2: rebase on top removed raid1
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-19 14:10:11 +03:00
Vitaly Baranov
16c8e93dd0
Merge pull request #36941 from nvartolomei/nv/show-grants-current-roles 
Allow a user to inspect grants from granted roles
 2022-05-19 09:20:52 +02:00
Alexander Tokmakov
e9568d8cbb fixes 2022-05-19 00:00:05 +02:00
Robert Schulze
a2214760c6
Remove mark_cache_size from further files 
- changed config.xml/yaml files used by CH's own internal tests which
  are (hopefully) not sensitive to mark_cache_size being set or not

- further occurrences exist but changing them seems a bad idea (e.g.
  because they are in customer-provided data)
 2022-05-18 22:09:14 +02:00
Alexander Tokmakov
9772924d06 Merge branch 'master' into fix_trash 2022-05-18 17:27:54 +02:00
Nikolai Kochetov
a19d4c6f1f
Merge pull request #37138 from azat/integration-tests-iptables 
tests/integration: fix possible race for iptables user rules inside containers
 2022-05-18 14:36:33 +02:00
Alexander Tokmakov
19afeda4b1 fix style 2022-05-18 14:23:52 +02:00
Nicolae Vartolomei
32b9959dec Allow a user to inspect grants from granted roles 2022-05-18 10:44:18 +00:00
alesapin
e99c90118e Merge branch 'master' into i_object_storage 2022-05-18 12:36:22 +02:00
lgbo-ustc
44d4ada542 fixed code style 2022-05-18 18:15:16 +08:00
Kruglov Pavel
492de1076c
Merge pull request #37232 from zzsmdfj/fix-mysql-datatype-binary0 
fix MySQL database engine to compatible with binary(0) dataType
 2022-05-18 12:15:07 +02:00
lgbo-ustc
9ed7f2109f fixed black check 2022-05-18 17:02:42 +08:00
lgbo-ustc
afed831785 fixed black check 2022-05-18 14:07:23 +08:00
lgbo-ustc
0578ef9372 add test case. test_hive_query::test_cache_dir_use 2022-05-18 10:26:12 +08:00
Alexander Tokmakov
dea39d8175 fix some trash 2022-05-17 18:22:52 +02:00
zzsmdfj
6a8913d2f6 to compatible with binary(0)-adjust code style 2022-05-17 23:08:40 +08:00
tchepavel
2692d88a42
Merge branch 'master' into nats-integration 2022-05-17 16:59:57 +03:00
tchepavel
a3af94d49e Documentation; integration tests; subscribe on startup 2022-05-17 16:58:09 +03:00
zzsmdfj
60ef84bf4c to compatible with binary(0)-fix code style 2022-05-17 20:15:59 +08:00
Vitaly Baranov
23a61b1596
Merge pull request #36700 from azat/fix-server-reload 
Fix server reload on port change (do not wait for current connections from query context)
 2022-05-17 12:25:21 +02:00
zzsmdfj
2ba1ca03f4 to compatible with binary(0)-add test 2022-05-17 15:30:38 +08:00
lgbo-ustc
319115cc63 update test case 2022-05-17 11:57:58 +08:00
Kseniia Sumarokova
94683786dc
Merge branch 'master' into MeiliSearch 2022-05-16 22:42:09 +02:00
Vitaly Baranov
72bff5242a
Merge pull request #37168 from vitlibar/backup-improvements-5 
Backups Improvements 5
 2022-05-16 21:25:54 +02:00
Azat Khuzhin
367daca584 Add a test for SYSTEM RELOAD CONFIG directly from the query context 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-16 16:22:58 +03:00
Azat Khuzhin
568e29ceb6 tests/integration/test_server_reload: add pylint hints 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-16 16:22:58 +03:00
Azat Khuzhin
b528a72e89 tests/integration/test_server_reload: add missing assertions 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-16 16:22:58 +03:00
Anton Ivashkin
191767d300 Reformat test file according to Check black 2022-05-16 12:35:21 +03:00
Vitaly Baranov
c1baad0763 Fix style. 2022-05-15 14:09:42 +02:00
alesapin
4e7e304e3d Fix BLACK 2022-05-14 18:53:30 +02:00
alesapin
61014dbde6 Simlify strange test 2022-05-14 18:41:54 +02:00
alesapin
4540cf8925 Style check fixes 2022-05-14 13:46:47 +02:00
alesapin
cc304a3178 Merge branch 'master' into i_object_storage 2022-05-13 23:44:00 +02:00
Mikhail Artemenko
c14fae24c4 run black for python files 2022-05-13 16:55:42 +03:00
Azat Khuzhin
b0c72b6104 tests/integration: add prefix match for skipped tests 
This way you can specify only file/module, or test name without
parameters.

Since, at least one, test that we care about
(test_distributed_respect_user_timeouts/test.py::test_reconnect) was not
runned sequentially [1].

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/37138/d7f5db4143c559bb8044058a653956945897e724/integration_tests__asan__actions__[2/3].html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-13 11:19:11 +03:00
Azat Khuzhin
f886c4237e tests/integration: fix possible race for iptables user rules inside containers 
It is possible for network PartitionManager to work incorrectly, because
of how docker setting up forward to DOCKER-USER chain, it first removes
forward and then adds it back (see [1] and [2]), however this introduce
race for a short period of time, and this is enough for TCP to
retransmit packets, and breaks network PartitionManager.

  [1]: b1e30e8328/libnetwork/iptables/iptables.go (L638)
  [2]: b1e30e8328/libnetwork/firewall_linux.go (L42)

Here are some details from logs for [3]:

    2022-04-27 03:01:00 [ 621 ] DEBUG : Executing query SELECT node FROM distributed_table ORDER BY node on node2 (cluster.py:2879, query_and_get_error)

  [3]: https://s3.amazonaws.com/clickhouse-test-reports/36295/314d553ab14d30df7508814513506ec09c7c7061/integration_tests__asan__actions__[2/3]/integration_run_parallel1_0.log

This query fails, from the server logs:

    2022.04.27 03:01:00.213101 [ 10 ] {19b1719f-8c39-4e3e-b782-aa4c933650f2} <Debug> executeQuery: (from 172.16.5.1:59008) SELECT node FROM distributed_table ORDER BY node
    ...
    2022.04.27 03:01:03.578439 [ 223 ] {19b1719f-8c39-4e3e-b782-aa4c933650f2} <Debug> Connection (node1:9000): Sent data for 2 scalars, total 2 rows in 0.000284672 sec., 6993 rows/sec., 68.00 B (232.15 KiB/sec.), compressed 0.4594594594594595 times to 148.00 B (505.16 KiB/sec.)
    2022.04.27 03:01:03.590637 [ 223 ] {19b1719f-8c39-4e3e-b782-aa4c933650f2} <Debug> MergingSortedTransform: Merge sorted 3 blocks, 2 rows in 3.371592744 sec., 0.5931914533744174 rows/sec., 94.61 B/sec
    2022.04.27 03:01:03.601256 [ 10 ] {19b1719f-8c39-4e3e-b782-aa4c933650f2} <Information> executeQuery: Read 2 rows, 28.00 B in 3.387950542 sec., 0 rows/sec., 8.26 B/sec.
    2022.04.27 03:01:03.601894 [ 10 ] {19b1719f-8c39-4e3e-b782-aa4c933650f2} <Debug> MemoryTracker: Peak memory usage (for query): 334.38 KiB.

And from docker daemon log:

    time="2022-04-27T03:00:59.916693113Z" level=debug msg="form data: {\"AttachStderr\":true,\"AttachStdin\":false,\"AttachStdout\":true,\"Cmd\":[\"iptables\",\"--wait\",\"-I\",\"DOCKER-USER\",\"1\",\"-p\",\"tcp\",\"-s\",\"172.16.5.2\",\"-d\",\"172.16.5.3\",\"-j\",\"DROP\"],\"Container\":\"b75f3b68cda51386bfbb9cceb67e92c4d217a5a1660bde2470b583cb1f4c7fc4\",\"Privileged\":true,\"Tty\":false,\"User\":\"\"}"
    time="2022-04-27T03:01:00.030654116Z" level=debug msg="form data: {\"AttachStderr\":true,\"AttachStdin\":false,\"AttachStdout\":true,\"Cmd\":[\"iptables\",\"--wait\",\"-I\",\"DOCKER-USER\",\"1\",\"-p\",\"tcp\",\"-s\",\"172.16.5.3\",\"-d\",\"172.16.5.2\",\"-j\",\"DROP\"],\"Container\":\"b75f3b68cda51386bfbb9cceb67e92c4d217a5a1660bde2470b583cb1f4c7fc4\",\"Privileged\":true,\"Tty\":false,\"User\":\"\"}"
    ...
    time="2022-04-27T03:01:03.515813984Z" level=debug msg="/usr/sbin/iptables, [--wait -t filter -n -L DOCKER-USER]"
    time="2022-04-27T03:01:03.531106486Z" level=debug msg="/usr/sbin/iptables, [--wait -t filter -C DOCKER-USER -j RETURN]"
    time="2022-04-27T03:01:03.535442346Z" level=debug msg="/usr/sbin/iptables, [--wait -t filter -C FORWARD -j DOCKER-USER]"
    time="2022-04-27T03:01:03.555856911Z" level=debug msg="/usr/sbin/iptables, [--wait -D FORWARD -j DOCKER-USER]"
    time="2022-04-27T03:01:03.564905764Z" level=debug msg="/usr/sbin/iptables, [--wait -I FORWARD -j DOCKER-USER]"
    ...
    time="2022-04-27T03:01:03.706374466Z" level=debug msg="form data: {\"AttachStderr\":true,\"AttachStdin\":false,\"AttachStdout\":true,\"Cmd\":[\"iptables\",\"--wait\",\"-D\",\"DOCKER-USER\",\"-p\",\"tcp\",\"-s\",\"172.16.5.3\",\"-d\",\"172.16.5.2\",\"-j\",\"DROP\"],\"Container\":\"b75f3b68cda51386bfbb9cceb67e92c4d217a5a1660bde2470b583cb1f4c7fc4\",\"Privileged\":true,\"Tty\":false,\"User\":\"\"}"
    time="2022-04-27T03:01:03.968077970Z" level=debug msg="form data: {\"AttachStderr\":true,\"AttachStdin\":false,\"AttachStdout\":true,\"Cmd\":[\"iptables\",\"--wait\",\"-D\",\"DOCKER-USER\",\"-p\",\"tcp\",\"-s\",\"172.16.5.2\",\"-d\",\"172.16.5.3\",\"-j\",\"DROP\"],\"Container\":\"b75f3b68cda51386bfbb9cceb67e92c4d217a5a1660bde2470b583cb1f4c7fc4\",\"Privileged\":true,\"Tty\":false,\"User\":\"\"}"

I've tried multiple ways of fixing this:

- Creating separate chain for rules from PartitionManager (DOCKER-USER-CLICKHOUSE)
  But it is created only once, and docker places new rules on top of the
  FORWARD chain, so it will not work, since it will not receive any
  packets

- Use DOCKER-USER, but replace iptables with a wrapper ([script]), that
  will ignore recreating of a rule for forward to DOCKER-USER, but this
  will not work too, since new docker rules will be created on top of
  FORWARD chain, and so DOCKER-USER will packets.

  [script]:

    if [[ "$*" =~ "-D FORWARD -j DOCKER-USER" ]]; then
        exit 0
    fi
    if [[ "$*" =~ "-I FORWARD -j DOCKER-USER" ]]; then
        if iptables.real iptables -C FORWARD -j DOCKER-USER; then
            exit 0
        fi
    fi

- And the only way to avoid flakiness for this case, is to forbid
  parallel execution for tests with PartitionManager.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-13 10:57:24 +03:00
Azat Khuzhin
9899b3f92d tests/integration: remove superfluous import of PartitionManager 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-13 10:57:24 +03:00
Azat Khuzhin
0fb9553e1c tests/integration: use no-resolve and verbose for iptables --list 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-13 10:57:24 +03:00
Azat Khuzhin
5c1b0f3a76 tests/integration: add logging for NetworkManager 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-13 10:57:24 +03:00
Vitaly Baranov
23322b0bf6 Add async tests. 2022-05-12 19:42:05 +02:00
Mikhail Artemenko
12717f2417 fix <Breaking changes> in Meilisearch v0.27.0 2022-05-12 16:51:00 +03:00
Vitaly Baranov
2c92fe21a9 Implement restoring to a bigger or smaller cluster. 
Remove backup settings allow_storing_multiple_replicas: now it's always allowed.
 2022-05-12 14:55:06 +02:00
Vitaly Baranov
30005a1eff BACKUP ON CLUSTER correctly collects data of a replicated table from all replicas now, 
and if some part doesn't exist on either replica it's no problem anymore.
 2022-05-12 13:33:42 +02:00
lgbo-ustc
5115fa8c6f update test case tests/integration/test_hive_query/test.py 2022-05-12 09:32:21 +08:00
alesapin
5dcd25be23 Initial implementation 2022-05-12 00:04:54 +02:00
mergify[bot]
f4bae3969d
Merge branch 'master' into enable-overcommit-for-user 2022-05-11 20:04:20 +00:00
Dmitry Novik
17608b3d93 Update documentation and defaults for memory overcommit 2022-05-11 16:18:41 +00:00
Kruglov Pavel
af3db5a992
Merge pull request #36726 from PolyProgrammist/s3clustermacro 
Handle cluster macro in s3Cluster table function
 2022-05-11 18:00:57 +02:00
mergify[bot]
0e2a86dcee
Merge branch 'master' into MeiliSearch 2022-05-11 08:49:19 +00:00
Mikhail f. Shiryaev
d5b1e39bb9
Merge pull request #36866 from ClickHouse/integration-tests 
Fix using proper image for integration tests; improve logging; fix docker volume cleaning in integration-tests-runner.
 2022-05-11 10:46:51 +02:00
Mikhail f. Shiryaev
76c3022029
Fix code broken in 1e82b04de3 2022-05-10 17:41:47 +02:00
Anton Popov
aec30c4076
Merge pull request #37053 from CurtizJ/remove-streams-comments 
Remove last mentions of data streams
 2022-05-10 13:38:13 +02:00
alesapin
41737d66b0
Merge pull request #37014 from ClickHouse/better_delete_outdated_parts 
Fix forgotten parts in cleanup thread
 2022-05-10 12:34:34 +02:00
Alexander Tokmakov
1d711496c3
Merge pull request #37035 from ClickHouse/Avogar-patch-4 
Fix flaky integration mongodb test
 2022-05-10 11:50:26 +03:00
Anton Popov
e911900054 remove last mentions of data streams 2022-05-09 19:15:24 +00:00
alesapin
068214c829 Turn black 2022-05-09 19:03:17 +02:00
Anton Ivashkin
6a6505c224 Merge master 2022-05-09 17:21:34 +03:00
mergify[bot]
8ea87a3530
Merge branch 'master' into enable-memory-overcommit 2022-05-09 14:00:54 +00:00
alesapin
018ed10684 Add test 2022-05-09 15:21:21 +02:00