e1ad5cee7e
pytest play games with logging output [1]. [1]: https://github.com/pytest-dev/pytest/issues/5502 But this does not actually affect ClickHouse tests, the only reason is that the kazoo client is not stopped correctly without calling kazoo.client.KazooClient.stop(), and that's why you can see the following messages: <details> test_storage_rabbitmq/test.py::test_rabbitmq_big_message --- Logging error --- Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/kazoo/protocol/connection.py", line 179, in _socket_error_handling yield File "/usr/local/lib/python3.8/dist-packages/kazoo/protocol/connection.py", line 659, in _connect self._socket = self.handler.create_connection( File "/usr/local/lib/python3.8/dist-packages/kazoo/handlers/threading.py", line 178, in create_connection return utils.create_tcp_connection(socket, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/kazoo/handlers/utils.py", line 265, in create_tcp_connection sock = module.create_connection(address, timeout_at) File "/usr/lib/python3.8/socket.py", line 808, in create_connection raise err File "/usr/lib/python3.8/socket.py", line 796, in create_connection sock.connect(sa) socket.timeout: timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/kazoo/protocol/connection.py", line 579, in _connect_attempt read_timeout, connect_timeout = self._connect(host, hostip, port) File "/usr/local/lib/python3.8/dist-packages/kazoo/protocol/connection.py", line 659, in _connect self._socket = self.handler.create_connection( File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/usr/local/lib/python3.8/dist-packages/kazoo/protocol/connection.py", line 182, in _socket_error_handling raise ConnectionDropped("socket connection error: %s" % (err,)) kazoo.exceptions.ConnectionDropped: socket connection error: None During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.8/logging/__init__.py", line 1088, in emit stream.write(msg + self.terminator) ValueError: I/O operation on closed file. Call stack: File "/usr/lib/python3.8/threading.py", line 890, in _bootstrap self._bootstrap_inner() File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner self.run() File "/usr/lib/python3.8/threading.py", line 870, in run self._target(*self._args, **self._kwargs) File "/usr/local/lib/python3.8/dist-packages/kazoo/protocol/connection.py", line 512, in zk_loop if retry(self._connect_loop, retry) is STOP_CONNECTING: File "/usr/local/lib/python3.8/dist-packages/kazoo/retry.py", line 126, in __call__ return func(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/kazoo/protocol/connection.py", line 552, in _connect_loop status = self._connect_attempt(host, hostip, port, retry) File "/usr/local/lib/python3.8/dist-packages/kazoo/protocol/connection.py", line 622, in _connect_attempt self.logger.warning('Connection dropped: %s', e) Message: 'Connection dropped: %s' Arguments: (ConnectionDropped('socket connection error: None'),) </details> Which eventually leads to incorrectly parsed report. This patch is an addition to the `logging.raiseExceptions=False` - #44618 Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com> |
||
---|---|---|
.. | ||
helpers | ||
test_access_control_on_cluster | ||
test_access_for_functions | ||
test_aggregation_memory_efficient | ||
test_allowed_client_hosts | ||
test_allowed_url_from_config | ||
test_alter_codec | ||
test_alter_on_mixed_type_cluster | ||
test_alter_settings_on_cluster | ||
test_alter_update_cast_keep_nullable | ||
test_always_fetch_merged | ||
test_async_drain_connection | ||
test_asynchronous_metric_log_table | ||
test_atomic_drop_table | ||
test_attach_backup_from_s3_plain | ||
test_attach_partition_with_large_destination | ||
test_attach_without_checksums | ||
test_attach_without_fetching | ||
test_authentication | ||
test_azure_blob_storage_zero_copy_replication | ||
test_backup_restore | ||
test_backup_restore_new | ||
test_backup_restore_on_cluster | ||
test_backup_restore_s3 | ||
test_backup_with_other_granularity | ||
test_backward_compatibility | ||
test_block_structure_mismatch | ||
test_broken_detached_part_clean_up | ||
test_broken_part_during_merge | ||
test_buffer_profile | ||
test_catboost_evaluate | ||
test_cgroup_limit | ||
test_check_table | ||
test_checking_s3_blobs_paranoid | ||
test_cleanup_after_start | ||
test_cleanup_dir_after_bad_zk_conn | ||
test_cluster_all_replicas | ||
test_cluster_copier | ||
test_cluster_discovery | ||
test_codec_encrypted | ||
test_composable_protocols | ||
test_compressed_marks_restart | ||
test_compression_codec_read | ||
test_compression_nested_columns | ||
test_concurrent_backups_s3 | ||
test_concurrent_queries_for_all_users_restriction | ||
test_concurrent_queries_for_user_restriction | ||
test_concurrent_queries_restriction_by_query_kind | ||
test_concurrent_threads_soft_limit | ||
test_concurrent_ttl_merges | ||
test_config_corresponding_root | ||
test_config_substitutions | ||
test_config_xml_full | ||
test_config_xml_main | ||
test_config_xml_yaml_mix | ||
test_config_yaml_full | ||
test_config_yaml_main | ||
test_consistant_parts_after_move_partition | ||
test_consistent_parts_after_clone_replica | ||
test_create_query_constraints | ||
test_create_user_and_login | ||
test_cross_replication | ||
test_custom_settings | ||
test_ddl_alter_query | ||
test_ddl_worker_non_leader | ||
test_default_compression_codec | ||
test_default_database_on_cluster | ||
test_default_role | ||
test_delayed_replica_failover | ||
test_detached_parts_metrics | ||
test_dictionaries_access | ||
test_dictionaries_all_layouts_separate_sources | ||
test_dictionaries_complex_key_cache_string | ||
test_dictionaries_config_reload | ||
test_dictionaries_ddl | ||
test_dictionaries_dependency | ||
test_dictionaries_dependency_xml | ||
test_dictionaries_mysql | ||
test_dictionaries_null_value | ||
test_dictionaries_postgresql | ||
test_dictionaries_redis | ||
test_dictionaries_select_all | ||
test_dictionaries_update_and_reload | ||
test_dictionaries_update_field | ||
test_dictionary_allow_read_expired_keys | ||
test_dictionary_custom_settings | ||
test_dictionary_ddl_on_cluster | ||
test_disabled_access_control_improvements | ||
test_disabled_mysql_server | ||
test_disk_access_storage | ||
test_disk_over_web_server | ||
test_disk_types | ||
test_disks_app_func | ||
test_distributed_backward_compatability | ||
test_distributed_ddl | ||
test_distributed_ddl_on_cross_replication | ||
test_distributed_ddl_parallel | ||
test_distributed_ddl_password | ||
test_distributed_default_database | ||
test_distributed_directory_monitor_split_batch_on_failure | ||
test_distributed_format | ||
test_distributed_insert_backward_compatibility | ||
test_distributed_inter_server_secret | ||
test_distributed_load_balancing | ||
test_distributed_over_distributed | ||
test_distributed_respect_user_timeouts | ||
test_distributed_storage_configuration | ||
test_distributed_system_query | ||
test_distributed_type_object | ||
test_dns_cache | ||
test_dotnet_client | ||
test_drop_is_lock_free | ||
test_drop_replica | ||
test_enabling_access_management | ||
test_encrypted_disk | ||
test_executable_dictionary | ||
test_executable_table_function | ||
test_executable_user_defined_function | ||
test_executable_user_defined_functions_config_reload | ||
test_explain_estimates | ||
test_extreme_deduplication | ||
test_failed_async_inserts | ||
test_fetch_memory_usage | ||
test_fetch_partition_from_auxiliary_zookeeper | ||
test_fetch_partition_should_reset_mutation | ||
test_fetch_partition_with_outdated_parts | ||
test_file_schema_inference_cache | ||
test_filesystem_layout | ||
test_force_deduplication | ||
test_force_drop_table | ||
test_format_avro_confluent | ||
test_format_schema_on_server | ||
test_freeze_table | ||
test_global_overcommit_tracker | ||
test_globs_in_filepath | ||
test_grant_and_revoke | ||
test_graphite_merge_tree | ||
test_graphite_merge_tree_typed | ||
test_groupBitmapAnd_on_distributed | ||
test_grpc_protocol | ||
test_grpc_protocol_ssl | ||
test_hedged_requests | ||
test_hedged_requests_parallel | ||
test_hive_query | ||
test_host_regexp_hosts_file_resolution | ||
test_host_regexp_multiple_ptr_records | ||
test_host_regexp_multiple_ptr_records_concurrent | ||
test_http_and_readonly | ||
test_http_handlers_config | ||
test_https_replication | ||
test_inherit_multiple_profiles | ||
test_input_format_parallel_parsing_memory_tracking | ||
test_insert_distributed_async_extra_dirs | ||
test_insert_distributed_async_send | ||
test_insert_distributed_load_balancing | ||
test_insert_into_distributed | ||
test_insert_into_distributed_sync_async | ||
test_insert_into_distributed_through_materialized_view | ||
test_inserts_with_keeper_retries | ||
test_jbod_balancer | ||
test_jbod_ha | ||
test_jbod_load_balancing | ||
test_jdbc_bridge | ||
test_jemalloc_percpu_arena | ||
test_join_set_family_s3 | ||
test_keeper_and_access_storage | ||
test_keeper_auth | ||
test_keeper_back_to_back | ||
test_keeper_force_recovery | ||
test_keeper_force_recovery_single_node | ||
test_keeper_four_word_command | ||
test_keeper_incorrect_config | ||
test_keeper_internal_secure | ||
test_keeper_map | ||
test_keeper_mntr_pressure | ||
test_keeper_multinode_blocade_leader | ||
test_keeper_multinode_simple | ||
test_keeper_nodes_add | ||
test_keeper_nodes_move | ||
test_keeper_nodes_remove | ||
test_keeper_persistent_log | ||
test_keeper_persistent_log_multinode | ||
test_keeper_restore_from_snapshot | ||
test_keeper_s3_snapshot | ||
test_keeper_secure_client | ||
test_keeper_session | ||
test_keeper_snapshot_on_exit | ||
test_keeper_snapshot_small_distance | ||
test_keeper_snapshots | ||
test_keeper_snapshots_multinode | ||
test_keeper_three_nodes_start | ||
test_keeper_three_nodes_two_alive | ||
test_keeper_two_nodes_cluster | ||
test_keeper_znode_time | ||
test_keeper_zookeeper_converter | ||
test_library_bridge | ||
test_limited_replicated_fetches | ||
test_log_family_hdfs | ||
test_log_family_s3 | ||
test_log_levels_update | ||
test_log_lz4_streaming | ||
test_log_query_probability | ||
test_logs_level | ||
test_lost_part | ||
test_lost_part_during_startup | ||
test_mask_sensitive_info | ||
test_match_process_uid_against_data_owner | ||
test_materialized_mysql_database | ||
test_materialized_view_restart_server | ||
test_max_http_connections_for_replication | ||
test_max_suspicious_broken_parts | ||
test_MemoryTracking | ||
test_merge_table_over_distributed | ||
test_merge_tree_azure_blob_storage | ||
test_merge_tree_empty_parts | ||
test_merge_tree_hdfs | ||
test_merge_tree_s3 | ||
test_merge_tree_s3_failover | ||
test_merge_tree_s3_restore | ||
test_merge_tree_s3_with_cache | ||
test_merge_tree_settings_constraints | ||
test_multiple_disks | ||
test_mutation_fetch_fallback | ||
test_mutations_hardlinks | ||
test_mutations_in_partitions_of_merge_tree | ||
test_mutations_with_merge_tree | ||
test_mutations_with_projection | ||
test_mysql_database_engine | ||
test_mysql_protocol | ||
test_named_collections | ||
test_non_default_compression | ||
test_odbc_interaction | ||
test_old_parts_finally_removed | ||
test_old_versions | ||
test_on_cluster_timeouts | ||
test_optimize_on_insert | ||
test_overcommit_tracker | ||
test_part_log_table | ||
test_part_moves_between_shards | ||
test_part_uuid | ||
test_partition | ||
test_parts_delete_zookeeper | ||
test_passing_max_partitions_to_read_remotely | ||
test_password_constraints | ||
test_polymorphic_parts | ||
test_postgresql_database_engine | ||
test_postgresql_protocol | ||
test_postgresql_replica_database_engine_1 | ||
test_postgresql_replica_database_engine_2 | ||
test_profile_events_s3 | ||
test_profile_settings_and_constraints_order | ||
test_prometheus_endpoint | ||
test_quorum_inserts | ||
test_quorum_inserts_parallel | ||
test_quota | ||
test_random_inserts | ||
test_range_hashed_dictionary_types | ||
test_read_only_table | ||
test_read_temporary_tables_on_failure | ||
test_recompression_ttl | ||
test_recovery_replica | ||
test_redirect_url_storage | ||
test_relative_filepath | ||
test_reload_auxiliary_zookeepers | ||
test_reload_certificate | ||
test_reload_clusters_config | ||
test_reload_max_table_size_to_drop | ||
test_reload_zookeeper | ||
test_reloading_settings_from_users_xml | ||
test_reloading_storage_configuration | ||
test_remote_prewhere | ||
test_rename_column | ||
test_replace_partition | ||
test_replica_can_become_leader | ||
test_replica_is_active | ||
test_replicated_database | ||
test_replicated_fetches_bandwidth | ||
test_replicated_fetches_timeouts | ||
test_replicated_merge_tree_compatibility | ||
test_replicated_merge_tree_config | ||
test_replicated_merge_tree_encrypted_disk | ||
test_replicated_merge_tree_encryption_codec | ||
test_replicated_merge_tree_hdfs_zero_copy | ||
test_replicated_merge_tree_s3 | ||
test_replicated_merge_tree_s3_restore | ||
test_replicated_merge_tree_s3_zero_copy | ||
test_replicated_merge_tree_with_auxiliary_zookeepers | ||
test_replicated_mutations | ||
test_replicated_parse_zk_metadata | ||
test_replicated_users | ||
test_replicating_constants | ||
test_replication_credentials | ||
test_replication_without_zookeeper | ||
test_restart_server | ||
test_restore_replica | ||
test_rocksdb_options | ||
test_rocksdb_read_only | ||
test_role | ||
test_row_policy | ||
test_s3_aws_sdk_has_slightly_unreliable_behaviour | ||
test_s3_cluster | ||
test_s3_low_cardinality_right_border | ||
test_s3_table_functions | ||
test_s3_with_https | ||
test_s3_with_proxy | ||
test_s3_zero_copy_replication | ||
test_s3_zero_copy_ttl | ||
test_secure_socket | ||
test_select_access_rights | ||
test_send_crash_reports | ||
test_send_request_to_leader_replica | ||
test_server_initialization | ||
test_server_reload | ||
test_server_start_and_ip_conversions | ||
test_settings_constraints | ||
test_settings_constraints_distributed | ||
test_settings_profile | ||
test_shard_level_const_function | ||
test_sharding_key_from_default_column | ||
test_sql_user_defined_functions_on_cluster | ||
test_ssl_cert_authentication | ||
test_storage_delta | ||
test_storage_dict | ||
test_storage_hdfs | ||
test_storage_hudi | ||
test_storage_kafka | ||
test_storage_kerberized_hdfs | ||
test_storage_kerberized_kafka | ||
test_storage_meilisearch | ||
test_storage_mongodb | ||
test_storage_mysql | ||
test_storage_nats | ||
test_storage_policies | ||
test_storage_postgresql | ||
test_storage_postgresql_replica | ||
test_storage_rabbitmq | ||
test_storage_s3 | ||
test_storage_url | ||
test_structured_logging_json | ||
test_system_clusters_actual_information | ||
test_system_ddl_worker_queue | ||
test_system_flush_logs | ||
test_system_logs_comment | ||
test_system_logs_recreate | ||
test_system_merges | ||
test_system_metrics | ||
test_system_queries | ||
test_system_replicated_fetches | ||
test_table_function_mongodb | ||
test_table_functions_access_rights | ||
test_tcp_handler_http_responses | ||
test_tcp_handler_interserver_listen_host | ||
test_temporary_data_in_cache | ||
test_text_log_level | ||
test_timezone_config | ||
test_tlsv1_3 | ||
test_tmp_policy | ||
test_transactions | ||
test_ttl_move | ||
test_ttl_replicated | ||
test_union_header | ||
test_user_defined_object_persistence | ||
test_user_directories | ||
test_user_ip_restrictions | ||
test_user_zero_database_access | ||
test_version_update | ||
test_version_update_after_mutation | ||
test_zookeeper_config | ||
test_zookeeper_config_load_balancing | ||
.dockerignore | ||
.gitignore | ||
ci-runner.py | ||
CMakeLists.txt | ||
conftest.py | ||
parallel_skip.json | ||
parallel.json | ||
parallel.readme | ||
pytest.ini | ||
README.md | ||
runner |
ClickHouse integration tests
This directory contains tests that involve several ClickHouse instances, custom configs, ZooKeeper, etc.
Running natively
Prerequisites:
- Ubuntu 20.04 (Focal) or higher.
- docker. Minimum required API version: 1.25, check with
docker version
.
You must install latest Docker from https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/#set-up-the-repository Don't use Docker from your system repository.
- pip and
libpq-dev
. To install:sudo apt-get install python3-pip libpq-dev zlib1g-dev libcrypto++-dev libssl-dev libkrb5-dev python3-dev
- py.test testing framework. To install:
sudo -H pip install pytest
- docker-compose and additional python libraries. To install:
sudo -H pip install \
PyMySQL \
aerospike \
avro \
cassandra-driver \
confluent-kafka \
dicttoxml \
docker \
docker-compose \
grpcio \
grpcio-tools \
kafka-python \
kazoo \
minio \
lz4 \
protobuf \
psycopg2-binary \
pymongo \
pytz \
pytest \
pytest-timeout \
redis \
tzlocal==2.1 \
urllib3 \
requests-kerberos \
dict2xml \
hypothesis \
pyhdfs \
pika \
meilisearch \
nats-py
(highly not recommended) If you really want to use OS packages on modern debian/ubuntu instead of "pip": sudo apt install -y docker docker-compose python3-pytest python3-dicttoxml python3-docker python3-pymysql python3-protobuf python3-pymongo python3-tzlocal python3-kazoo python3-psycopg2 kafka-python python3-pytest-timeout python3-minio
If you want to run the tests under a non-privileged user, you must add this user to docker
group: sudo usermod -aG docker $USER
and re-login.
(You must close all your sessions (for example, restart your computer))
To check, that you have access to Docker, run docker ps
.
Run the tests with the pytest
command. To select which tests to run, use: pytest -k <test_name_pattern>
By default tests are run with system-wide client binary, server binary and base configs. To change that, set the following environment variables:
CLICKHOUSE_TESTS_SERVER_BIN_PATH
to choose the server binary.CLICKHOUSE_TESTS_CLIENT_BIN_PATH
to choose the client binary.CLICKHOUSE_TESTS_BASE_CONFIG_DIR
to choose the directory from which base configs (config.xml
andusers.xml
) are taken.
Please note that if you use separate build (ENABLE_CLICKHOUSE_ALL=OFF
), you need to build different components, including but not limited to ENABLE_CLICKHOUSE_LIBRARY_BRIDGE=ON ENABLE_CLICKHOUSE_ODBC_BRIDGE=ON ENABLE_CLICKHOUSE_KEEPER=ON
. So it is easier to use ENABLE_CLICKHOUSE_ALL=ON
For tests that use common docker compose files you may need to set up their path with environment variable: DOCKER_COMPOSE_DIR=$HOME/ClickHouse/docker/test/integration/runner/compose
Running with runner script
The only requirement is fresh configured docker and docker pull clickhouse/integration-tests-runner
Notes:
- If you want to run integration tests without
sudo
you have to add your user to docker groupsudo usermod -aG docker $USER
. More information about docker configuration. - If you already had run these tests without
./runner
script you may have problems with pytest cache. It can be removed withrm -r __pycache__ .pytest_cache/
. - Some tests maybe require a lot of resources (CPU, RAM, etc.). Better not try large tests like
test_cluster_copier
ortest_distributed_ddl*
on your laptop.
You can run tests via ./runner
script and pass pytest arguments as last arg:
$ ./runner --binary $HOME/ClickHouse/programs/clickhouse --odbc-bridge-binary $HOME/ClickHouse/programs/clickhouse-odbc-bridge --base-configs-dir $HOME/ClickHouse/programs/server/ 'test_ssl_cert_authentication -ss'
Start tests
====================================================================================================== test session starts ======================================================================================================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /ClickHouse/tests/integration, configfile: pytest.ini
plugins: repeat-0.9.1, xdist-2.5.0, forked-1.4.0, order-1.0.0, timeout-2.1.0
timeout: 900.0s
timeout method: signal
timeout func_only: False
collected 4 items
test_ssl_cert_authentication/test.py::test_https Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml
PASSED
test_ssl_cert_authentication/test.py::test_https_wrong_cert PASSED
test_ssl_cert_authentication/test.py::test_https_non_ssl_auth PASSED
test_ssl_cert_authentication/test.py::test_create_user PASSED
================================================================================================= 4 passed in 118.58s (0:01:58) =================================================================================================
Path to binary and configs maybe specified via env variables:
$ export CLICKHOUSE_TESTS_BASE_CONFIG_DIR=$HOME/ClickHouse/programs/server/
$ export CLICKHOUSE_TESTS_SERVER_BIN_PATH=$HOME/ClickHouse/programs/clickhouse
$ export CLICKHOUSE_TESTS_ODBC_BRIDGE_BIN_PATH=$HOME/ClickHouse/programs/clickhouse-odbc-bridge
$ ./runner 'test_odbc_interaction'
$ # or ./runner '-v -ss'
Start tests
============================= test session starts ==============================
platform linux2 -- Python 2.7.15rc1, pytest-4.0.0, py-1.7.0, pluggy-0.8.0
rootdir: /ClickHouse/tests/integration, inifile: pytest.ini
collected 6 items
test_odbc_interaction/test.py ...... [100%]
==================== 6 passed, 1 warnings in 96.33 seconds =====================
You can just open shell inside a container by overwritting the command: ./runner --command=bash
Rebuilding the docker containers
The main container used for integration tests lives in docker/test/integration/base/Dockerfile
. Rebuild it with
cd docker/test/integration/base
docker build -t clickhouse/integration-test .
The helper container used by the runner
script is in docker/test/integration/runner/Dockerfile
.
Adding new tests
To add new test named foo
, create a directory test_foo
with an empty __init__.py
and a file
named test.py
containing tests in it. All functions with names starting with test
will become test cases.
helpers
directory contains utilities for:
- Launching a ClickHouse cluster with or without ZooKeeper in docker containers.
- Sending queries to launched instances.
- Introducing network failures such as severing network link between two instances.
To assert that two TSV files must be equal, wrap them in the TSV
class and use the regular assert
statement. Example: assert TSV(result) == TSV(reference)
. In case the assertion fails, pytest
will automagically detect the types of variables and only the small diff of two files is printed.
Troubleshooting
If tests failing for mysterious reasons, this may help:
sudo service docker stop
sudo bash -c 'rm -rf /var/lib/docker/*'
sudo service docker start
iptables-nft
On Ubuntu 20.10 and later in host network mode (default) one may encounter problem with nested containers not seeing each other. It happens because legacy and nftables rules are out of sync. Problem can be solved by:
sudo iptables -P FORWARD ACCEPT