runner is used by developers to run tests, while ci-runner.py is used
only by CI scripts, and to avoid requiring CAP_SYSLOG for manual dmesg
clear/dump had been moved.
Also for manual runs this can be done manually.
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
pytest play games with logging output [1].
[1]: https://github.com/pytest-dev/pytest/issues/5502
But this does not actually affect ClickHouse tests, the only reason is
that the kazoo client is not stopped correctly without calling
kazoo.client.KazooClient.stop(), and that's why you can see the
following messages:
<details>
test_storage_rabbitmq/test.py::test_rabbitmq_big_message --- Logging error ---
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/kazoo/protocol/connection.py", line 179, in _socket_error_handling
yield
File "/usr/local/lib/python3.8/dist-packages/kazoo/protocol/connection.py", line 659, in _connect
self._socket = self.handler.create_connection(
File "/usr/local/lib/python3.8/dist-packages/kazoo/handlers/threading.py", line 178, in create_connection
return utils.create_tcp_connection(socket, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/kazoo/handlers/utils.py", line 265, in create_tcp_connection
sock = module.create_connection(address, timeout_at)
File "/usr/lib/python3.8/socket.py", line 808, in create_connection
raise err
File "/usr/lib/python3.8/socket.py", line 796, in create_connection
sock.connect(sa)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/kazoo/protocol/connection.py", line 579, in _connect_attempt
read_timeout, connect_timeout = self._connect(host, hostip, port)
File "/usr/local/lib/python3.8/dist-packages/kazoo/protocol/connection.py", line 659, in _connect
self._socket = self.handler.create_connection(
File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/local/lib/python3.8/dist-packages/kazoo/protocol/connection.py", line 182, in _socket_error_handling
raise ConnectionDropped("socket connection error: %s" % (err,))
kazoo.exceptions.ConnectionDropped: socket connection error: None
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.8/logging/__init__.py", line 1088, in emit
stream.write(msg + self.terminator)
ValueError: I/O operation on closed file.
Call stack:
File "/usr/lib/python3.8/threading.py", line 890, in _bootstrap
self._bootstrap_inner()
File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
File "/usr/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.8/dist-packages/kazoo/protocol/connection.py", line 512, in zk_loop
if retry(self._connect_loop, retry) is STOP_CONNECTING:
File "/usr/local/lib/python3.8/dist-packages/kazoo/retry.py", line 126, in __call__
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/kazoo/protocol/connection.py", line 552, in _connect_loop
status = self._connect_attempt(host, hostip, port, retry)
File "/usr/local/lib/python3.8/dist-packages/kazoo/protocol/connection.py", line 622, in _connect_attempt
self.logger.warning('Connection dropped: %s', e)
Message: 'Connection dropped: %s'
Arguments: (ConnectionDropped('socket connection error: None'),)
</details>
Which eventually leads to incorrectly parsed report.
This patch is an addition to the `logging.raiseExceptions=False` - #44618
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Right now BACKUP omit similar files, and will not allow to use this
backup as a regular table, and usually those similar files are quite
small (i.e. columns.txt).
So by using `BACKUP TO S3() deduplicate_files=0` you will be possible to
use `ATTACH TABLE` directly from S3.
P.S. right now it is possible only for the table with one part, since,
usually, there is nothing to deduplicate (if the columns are different).
v2: Add deduplicate_files into metadata
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Due to pytest play games with logging [1] it is better to ignore it.
[1]: https://github.com/pytest-dev/pytest/issues/5502
This is due to incorrect kazoo client usage (not all clients calls
stop())
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
Implementation:
* Updated to substitute macros in endpoint for all disks (s3, hfs, web).
Testing:
* Added an integration test test/integration/test_endpoint_macro_substitution.
