thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-09-21 01:00:48 +00:00
Code Issues Packages Projects Releases Wiki Activity
89,003 Commits 8,935 Branches 1,766 Tags 2 GiB
f69c3175af
Commit Graph

4205 Commits

Author SHA1 Message Date
Nikolai Kochetov
a19d4c6f1f
Merge pull request #37138 from azat/integration-tests-iptables 
tests/integration: fix possible race for iptables user rules inside containers
 2022-05-18 14:36:33 +02:00
Kruglov Pavel
492de1076c
Merge pull request #37232 from zzsmdfj/fix-mysql-datatype-binary0 
fix MySQL database engine to compatible with binary(0) dataType
 2022-05-18 12:15:07 +02:00
zzsmdfj
6a8913d2f6 to compatible with binary(0)-adjust code style 2022-05-17 23:08:40 +08:00
zzsmdfj
60ef84bf4c to compatible with binary(0)-fix code style 2022-05-17 20:15:59 +08:00
Vitaly Baranov
23a61b1596
Merge pull request #36700 from azat/fix-server-reload 
Fix server reload on port change (do not wait for current connections from query context)
 2022-05-17 12:25:21 +02:00
zzsmdfj
2ba1ca03f4 to compatible with binary(0)-add test 2022-05-17 15:30:38 +08:00
Kseniia Sumarokova
94683786dc
Merge branch 'master' into MeiliSearch 2022-05-16 22:42:09 +02:00
Vitaly Baranov
72bff5242a
Merge pull request #37168 from vitlibar/backup-improvements-5 
Backups Improvements 5
 2022-05-16 21:25:54 +02:00
Azat Khuzhin
367daca584 Add a test for SYSTEM RELOAD CONFIG directly from the query context 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-16 16:22:58 +03:00
Azat Khuzhin
568e29ceb6 tests/integration/test_server_reload: add pylint hints 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-16 16:22:58 +03:00
Azat Khuzhin
b528a72e89 tests/integration/test_server_reload: add missing assertions 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-16 16:22:58 +03:00
Vitaly Baranov
c1baad0763 Fix style. 2022-05-15 14:09:42 +02:00
Mikhail Artemenko
c14fae24c4 run black for python files 2022-05-13 16:55:42 +03:00
Azat Khuzhin
b0c72b6104 tests/integration: add prefix match for skipped tests 
This way you can specify only file/module, or test name without
parameters.

Since, at least one, test that we care about
(test_distributed_respect_user_timeouts/test.py::test_reconnect) was not
runned sequentially [1].

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/37138/d7f5db4143c559bb8044058a653956945897e724/integration_tests__asan__actions__[2/3].html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-13 11:19:11 +03:00
Azat Khuzhin
f886c4237e tests/integration: fix possible race for iptables user rules inside containers 
It is possible for network PartitionManager to work incorrectly, because
of how docker setting up forward to DOCKER-USER chain, it first removes
forward and then adds it back (see [1] and [2]), however this introduce
race for a short period of time, and this is enough for TCP to
retransmit packets, and breaks network PartitionManager.

  [1]: b1e30e8328/libnetwork/iptables/iptables.go (L638)
  [2]: b1e30e8328/libnetwork/firewall_linux.go (L42)

Here are some details from logs for [3]:

    2022-04-27 03:01:00 [ 621 ] DEBUG : Executing query SELECT node FROM distributed_table ORDER BY node on node2 (cluster.py:2879, query_and_get_error)

  [3]: https://s3.amazonaws.com/clickhouse-test-reports/36295/314d553ab14d30df7508814513506ec09c7c7061/integration_tests__asan__actions__[2/3]/integration_run_parallel1_0.log

This query fails, from the server logs:

    2022.04.27 03:01:00.213101 [ 10 ] {19b1719f-8c39-4e3e-b782-aa4c933650f2} <Debug> executeQuery: (from 172.16.5.1:59008) SELECT node FROM distributed_table ORDER BY node
    ...
    2022.04.27 03:01:03.578439 [ 223 ] {19b1719f-8c39-4e3e-b782-aa4c933650f2} <Debug> Connection (node1:9000): Sent data for 2 scalars, total 2 rows in 0.000284672 sec., 6993 rows/sec., 68.00 B (232.15 KiB/sec.), compressed 0.4594594594594595 times to 148.00 B (505.16 KiB/sec.)
    2022.04.27 03:01:03.590637 [ 223 ] {19b1719f-8c39-4e3e-b782-aa4c933650f2} <Debug> MergingSortedTransform: Merge sorted 3 blocks, 2 rows in 3.371592744 sec., 0.5931914533744174 rows/sec., 94.61 B/sec
    2022.04.27 03:01:03.601256 [ 10 ] {19b1719f-8c39-4e3e-b782-aa4c933650f2} <Information> executeQuery: Read 2 rows, 28.00 B in 3.387950542 sec., 0 rows/sec., 8.26 B/sec.
    2022.04.27 03:01:03.601894 [ 10 ] {19b1719f-8c39-4e3e-b782-aa4c933650f2} <Debug> MemoryTracker: Peak memory usage (for query): 334.38 KiB.

And from docker daemon log:

    time="2022-04-27T03:00:59.916693113Z" level=debug msg="form data: {\"AttachStderr\":true,\"AttachStdin\":false,\"AttachStdout\":true,\"Cmd\":[\"iptables\",\"--wait\",\"-I\",\"DOCKER-USER\",\"1\",\"-p\",\"tcp\",\"-s\",\"172.16.5.2\",\"-d\",\"172.16.5.3\",\"-j\",\"DROP\"],\"Container\":\"b75f3b68cda51386bfbb9cceb67e92c4d217a5a1660bde2470b583cb1f4c7fc4\",\"Privileged\":true,\"Tty\":false,\"User\":\"\"}"
    time="2022-04-27T03:01:00.030654116Z" level=debug msg="form data: {\"AttachStderr\":true,\"AttachStdin\":false,\"AttachStdout\":true,\"Cmd\":[\"iptables\",\"--wait\",\"-I\",\"DOCKER-USER\",\"1\",\"-p\",\"tcp\",\"-s\",\"172.16.5.3\",\"-d\",\"172.16.5.2\",\"-j\",\"DROP\"],\"Container\":\"b75f3b68cda51386bfbb9cceb67e92c4d217a5a1660bde2470b583cb1f4c7fc4\",\"Privileged\":true,\"Tty\":false,\"User\":\"\"}"
    ...
    time="2022-04-27T03:01:03.515813984Z" level=debug msg="/usr/sbin/iptables, [--wait -t filter -n -L DOCKER-USER]"
    time="2022-04-27T03:01:03.531106486Z" level=debug msg="/usr/sbin/iptables, [--wait -t filter -C DOCKER-USER -j RETURN]"
    time="2022-04-27T03:01:03.535442346Z" level=debug msg="/usr/sbin/iptables, [--wait -t filter -C FORWARD -j DOCKER-USER]"
    time="2022-04-27T03:01:03.555856911Z" level=debug msg="/usr/sbin/iptables, [--wait -D FORWARD -j DOCKER-USER]"
    time="2022-04-27T03:01:03.564905764Z" level=debug msg="/usr/sbin/iptables, [--wait -I FORWARD -j DOCKER-USER]"
    ...
    time="2022-04-27T03:01:03.706374466Z" level=debug msg="form data: {\"AttachStderr\":true,\"AttachStdin\":false,\"AttachStdout\":true,\"Cmd\":[\"iptables\",\"--wait\",\"-D\",\"DOCKER-USER\",\"-p\",\"tcp\",\"-s\",\"172.16.5.3\",\"-d\",\"172.16.5.2\",\"-j\",\"DROP\"],\"Container\":\"b75f3b68cda51386bfbb9cceb67e92c4d217a5a1660bde2470b583cb1f4c7fc4\",\"Privileged\":true,\"Tty\":false,\"User\":\"\"}"
    time="2022-04-27T03:01:03.968077970Z" level=debug msg="form data: {\"AttachStderr\":true,\"AttachStdin\":false,\"AttachStdout\":true,\"Cmd\":[\"iptables\",\"--wait\",\"-D\",\"DOCKER-USER\",\"-p\",\"tcp\",\"-s\",\"172.16.5.2\",\"-d\",\"172.16.5.3\",\"-j\",\"DROP\"],\"Container\":\"b75f3b68cda51386bfbb9cceb67e92c4d217a5a1660bde2470b583cb1f4c7fc4\",\"Privileged\":true,\"Tty\":false,\"User\":\"\"}"

I've tried multiple ways of fixing this:

- Creating separate chain for rules from PartitionManager (DOCKER-USER-CLICKHOUSE)
  But it is created only once, and docker places new rules on top of the
  FORWARD chain, so it will not work, since it will not receive any
  packets

- Use DOCKER-USER, but replace iptables with a wrapper ([script]), that
  will ignore recreating of a rule for forward to DOCKER-USER, but this
  will not work too, since new docker rules will be created on top of
  FORWARD chain, and so DOCKER-USER will packets.

  [script]:

    if [[ "$*" =~ "-D FORWARD -j DOCKER-USER" ]]; then
        exit 0
    fi
    if [[ "$*" =~ "-I FORWARD -j DOCKER-USER" ]]; then
        if iptables.real iptables -C FORWARD -j DOCKER-USER; then
            exit 0
        fi
    fi

- And the only way to avoid flakiness for this case, is to forbid
  parallel execution for tests with PartitionManager.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-13 10:57:24 +03:00
Azat Khuzhin
9899b3f92d tests/integration: remove superfluous import of PartitionManager 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-13 10:57:24 +03:00
Azat Khuzhin
0fb9553e1c tests/integration: use no-resolve and verbose for iptables --list 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-13 10:57:24 +03:00
Azat Khuzhin
5c1b0f3a76 tests/integration: add logging for NetworkManager 
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2022-05-13 10:57:24 +03:00
Vitaly Baranov
23322b0bf6 Add async tests. 2022-05-12 19:42:05 +02:00
Mikhail Artemenko
12717f2417 fix <Breaking changes> in Meilisearch v0.27.0 2022-05-12 16:51:00 +03:00
Vitaly Baranov
2c92fe21a9 Implement restoring to a bigger or smaller cluster. 
Remove backup settings allow_storing_multiple_replicas: now it's always allowed.
 2022-05-12 14:55:06 +02:00
Vitaly Baranov
30005a1eff BACKUP ON CLUSTER correctly collects data of a replicated table from all replicas now, 
and if some part doesn't exist on either replica it's no problem anymore.
 2022-05-12 13:33:42 +02:00
mergify[bot]
f4bae3969d
Merge branch 'master' into enable-overcommit-for-user 2022-05-11 20:04:20 +00:00
Dmitry Novik
17608b3d93 Update documentation and defaults for memory overcommit 2022-05-11 16:18:41 +00:00
Kruglov Pavel
af3db5a992
Merge pull request #36726 from PolyProgrammist/s3clustermacro 
Handle cluster macro in s3Cluster table function
 2022-05-11 18:00:57 +02:00
mergify[bot]
0e2a86dcee
Merge branch 'master' into MeiliSearch 2022-05-11 08:49:19 +00:00
Mikhail f. Shiryaev
d5b1e39bb9
Merge pull request #36866 from ClickHouse/integration-tests 
Fix using proper image for integration tests; improve logging; fix docker volume cleaning in integration-tests-runner.
 2022-05-11 10:46:51 +02:00
Mikhail f. Shiryaev
76c3022029
Fix code broken in 1e82b04de3 2022-05-10 17:41:47 +02:00
Anton Popov
aec30c4076
Merge pull request #37053 from CurtizJ/remove-streams-comments 
Remove last mentions of data streams
 2022-05-10 13:38:13 +02:00
alesapin
41737d66b0
Merge pull request #37014 from ClickHouse/better_delete_outdated_parts 
Fix forgotten parts in cleanup thread
 2022-05-10 12:34:34 +02:00
Alexander Tokmakov
1d711496c3
Merge pull request #37035 from ClickHouse/Avogar-patch-4 
Fix flaky integration mongodb test
 2022-05-10 11:50:26 +03:00
Anton Popov
e911900054 remove last mentions of data streams 2022-05-09 19:15:24 +00:00
alesapin
068214c829 Turn black 2022-05-09 19:03:17 +02:00
mergify[bot]
8ea87a3530
Merge branch 'master' into enable-memory-overcommit 2022-05-09 14:00:54 +00:00
alesapin
018ed10684 Add test 2022-05-09 15:21:21 +02:00
Alexander Tokmakov
b6be194d3a
Merge pull request #36976 from ClickHouse/replicated_database_fixes 
Fixes for Replicated database
 2022-05-09 14:41:49 +03:00
Kruglov Pavel
23bd9390a6
Merge pull request #36997 from vitlibar/users-without-assigned-policies-can-view-rows 
Users without assigned row policies can view rows
 2022-05-09 12:04:50 +02:00
Kruglov Pavel
31c56c95c8
Fix flaky integration mongodb test 2022-05-09 12:04:05 +02:00
Alexey Milovidov
3755466e8d
Merge pull request #34916 from nic11/pr/content-type 
Support content_type in HTTP handler config
 2022-05-08 16:36:21 +03:00
Vitaly Baranov
1b2eb4fe27 Use more clear syntax for BACKUP/RESTORE. 2022-05-08 10:37:02 +02:00
Vitaly Baranov
69bec2f377 Users without assigned row policies can view rows now. 2022-05-07 14:50:24 +02:00
Dmitry Novik
46fa87b14e Fix tests 2022-05-06 16:46:54 +00:00
Alexander Tokmakov
c806869040 fix sync database replica 2022-05-06 18:37:20 +02:00
mergify[bot]
96fb6f7b92
Merge branch 'master' into fix-checks-in-tests 2022-05-05 15:43:27 +00:00
Vitaly Baranov
dd51265218
Merge pull request #36864 from vitlibar/backup-improvements-4 
Backup improvements
 2022-05-05 15:37:51 +02:00
tchepavel
e09a59ae39 Fix checking for rabbitmq liveness in tests 2022-05-05 15:25:42 +03:00
Alexander Gololobov
7e3a805ae3
Merge pull request #36572 from ClickHouse/fix-trash-s3-metrics 
Everything related to S3 metrics was not good
 2022-05-05 11:44:02 +02:00
mergify[bot]
e0fa563092
Merge branch 'master' into fix_exception_message 2022-05-04 15:56:03 +00:00
mergify[bot]
6ecddf4077
Merge branch 'master' into fix-trash-s3-metrics 2022-05-04 10:14:29 +00:00
Alexander Tokmakov
ea9c697348 disable broken tests 2022-05-04 11:45:36 +02:00