Sometimes you may get:
> raise subprocess.CalledProcessError(exit_code, cmd)
E subprocess.CalledProcessError: Command '['iptables', '--wait', '-D', 'DOCKER-USER', '-p', 'tcp', '-s', '172.16.2.3', '-d', '172.16.2.2', '-j', 'DROP']' returned non-zero exit status 137.
And only sometimes you may get the reason:
OCI runtime exec failed: exec failed: cannot exec in a stopped container: unknown
So this means that container for iptables does not exists anymore, and
the reason is the timeout. And the fact that container_exit_timeout was
equal to container_expire_timeout and was 120.
From the docker logs:
time="2023-07-16T15:46:52.513673446Z" level=debug msg="form data: {\"AttachStderr\":false,\"AttachStdin\":false,\"AttachStdout\":false,\"Cmd\":[\"sleep\",\"120\"],\"HostConfig\":{\"AutoRemove\":true,\"NetworkMode\":\"host\"},\"Image\":\"clickhouse/integration-helper:latest\",\"NetworkDisabled\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Tty\":false}"
time="2023-07-16T15:48:57.611857183Z" level=debug msg="form data: {\"AttachStderr\":false,\"AttachStdin\":false,\"AttachStdout\":false,\"Cmd\":[\"sleep\",\"120\"],\"HostConfig\":{\"AutoRemove\":true,\"NetworkMode\":\"host\"},\"Image\":\"clickhouse/integration-helper:latest\",\"NetworkDisabled\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Tty\":false}"
And then pytest will try to execute the iptables command:
time="2023-07-16T15:50:57.698705244Z" level=debug msg="starting exec command 860920ab2aa07e8d285050f200ac92423a3cf8ec3fb2f57683541e62cf6bc20e in container 66d6c96671b5e987345290ddd260727d96b99789b512d40f333f6263f42fd2f1"
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
This is part of **"Why my PR is not merged?"** patch set series.
Since #50107 there one more tunable that can increase the cleanup period
- max_cleanup_delay_period.
Plus, increase the delay for the cleanup thread, otherwise it is too
fragile.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
At first, there was no such amount of clusters in dist config, they
added when someone need to write some new cluster for tests.
So let's move them to the clusters.xml that is deployed only for tests,
and leave only default cluster.
And cleanup also some configs that had been copied from dist config in
the repo (about test_config_* integration tests, this should be OK,
since there are more_clusters.xml as well, that covers additional
cases).
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
