ClickHouse/tests/testflows
2021-05-09 18:29:33 +03:00
..
aes_encryption Fixing retention precision for failing RBAC tests. 2021-03-10 07:22:18 -05:00
datetime64_extended_range Update date_time_functions.py 2021-04-14 14:09:51 +03:00
example Fixing retention precision for failing RBAC tests. 2021-03-10 07:22:18 -05:00
helpers All tests ready 2021-04-11 01:30:48 +03:00
kerberos Add Kerberos support for authenticating existing users when accessing over HTTP 2021-03-11 23:41:10 +03:00
ldap Merge pull request #21551 from vzakaznikov/testflows_use_yandex_docker_repo 2021-03-22 02:27:50 +03:00
map_type Crossing out failing map type tests. 2021-04-21 16:29:41 -04:00
rbac fix suggests 2021-04-14 10:14:13 +08:00
window_functions Fixing window functions distributed tests by moving to using deterministic sharding key. 2021-05-08 15:06:31 -04:00
README.md Small fix to the test/testflows/README.md 2020-12-04 19:27:41 -05:00
regression.py Merge branch 'master' into fixing_testflows_window_function_distributed_tests 2021-05-09 18:29:33 +03:00
runner fix 2021-02-27 14:01:04 +03:00

ClickHouse Tests in TestFlows

This directory contains integration tests written using TestFlows that involves several ClickHouse instances, custom configs, ZooKeeper, etc.

Supported environment

Prerequisites

Running tests locally

You can run tests locally by passing --local and --clickhouse-binary-path to the regression.py.

  • --local specifies that regression will be run locally
  • --clickhouse-binary-path specifies the path to the ClickHouse binary that will be used during the regression run

Note: you can pass -h or --help argument to the regression.py to see a help message.

python3 regression.py -h

Note: make sure that the ClickHouse binary has correct permissions. If you are using /usr/bin/clickhouse its owner and group is set to root:root by default and it needs to be changed to clickhouse:clickhouse. You can change the owner and the group using the following command.

sudo chown clickhouse:clickhouse /usr/bin/clickhouse

Using the default ClickHouse installation and its server binary at /usr/bin/clickhouse, you can run regressions locally using the following command.

python3 regression.py --local --clickhouse-binary-path "/usr/bin/clickhouse"

Output Verbosity

You can control verbosity of the output by specifying the output format with -o or --output option. See --help for more details.

Running Only Selected Tests

You can run only the selected tests by passing --only option to the regression.py.

For example,

./regression.py --local --clickhouse-binary-path /usr/bin/clickhouse --only "/clickhouse/rbac/syntax/grant privilege/*"

will execute all rbac/syntax/grant privilege tests.

If you want to run only a single test such as the /clickhouse/rbac/syntax/grant privilege/grant privileges/privilege='KILL QUERY', on=('*.*',), allow_introspection=False you can do it as follows

./regression.py --local --clickhouse-binary-path /usr/bin/clickhouse --only "/clickhouse/rbac/syntax/grant privilege/grant privileges/privilege='KILL QUERY', on=('[*].[*]',), allow_introspection=False/*"

Note that you need to surround special characters such as * with square brackets, for example [*].

Note that you need to end the filtering pattern with /* to run all the steps inside the test.

For more information, please see Filtering section in the TestFlows Handbook.

How To Debug Why Test Failed

Step 1: find which tests failed

If TestFlows check does not pass you should look at the end of the test_run.txt.out.log to find the list of failing tests. For example,

clickhouse_testflows_tests_volume
Start tests
➤ Dec 02,2020 22:22:24 /clickhouse
...
Failing

✘ [ Fail ] /clickhouse/rbac/syntax/grant privilege/grant privileges/privilege='SELECT', on=('db0.table0', 'db0.*', '*.*', 'tb0', '*'), allow_column=True, allow_introspection=False
✘ [ Fail ] /clickhouse/rbac/syntax/grant privilege/grant privileges
✘ [ Fail ] /clickhouse/rbac/syntax/grant privilege
✘ [ Fail ] /clickhouse/rbac/syntax
✘ [ Fail ] /clickhouse/rbac
✘ [ Fail ] /clickhouse

In this case the failing test is

/clickhouse/rbac/syntax/grant privilege/grant privileges/privilege='SELECT', on=('db0.table0', 'db0.*', '*.*', 'tb0', '*'), allow_column=True, allow_introspection=False

while the others

✘ [ Fail ] /clickhouse/rbac/syntax/grant privilege/grant privileges
✘ [ Fail ] /clickhouse/rbac/syntax/grant privilege
✘ [ Fail ] /clickhouse/rbac/syntax
✘ [ Fail ] /clickhouse/rbac
✘ [ Fail ] /clickhouse

failed because the first fail gets "bubble-up" the test execution tree all the way to the top level test which is the /clickhouse.

Step 2: download test.log that contains all raw messages

You need to download the test.log that contains all raw messages.

Step 3: get messages for the failing test

Once you know the name of the failing test and you have the test.log that contains all the raw messages for all the tests, you can use tfs show test messages command.

You get the tfs command by installing TestFlows.

For example,

cat test.log | tfs show test messages "/clickhouse/rbac/syntax/grant privilege/grant privileges/privilege='SELECT', on=\('db0.table0', 'db0.\*', '\*.\*', 'tb0', '\*'\), allow_column=True, allow_introspection=False"

Note: that characters that are treated as special in extended regular expressions need to be escaped. In this case we have to escape the *, (, and the ) characters in the test name.

Step 4: working with the test.log

You can use the test.log with many of the commands provided by the tfs utility.

See tfs --help for more information.

For example, you can get a list of failing tests from the test.log using the tfs show fails command as follows

$ cat test.log | tfs show fails

or get the results using the tfs show results command as follows

$ cat test.log | tfs show results

or you can transform the log to see only the new fails using the tfs transform fail --new command as follows

$ cat test.log | tfs transform fails --new