mirror of
https://github.com/ClickHouse/ClickHouse.git
synced 2024-11-24 00:22:29 +00:00
Updated documentation about ClickHouse testing [#CLICKHOUSE-2]
This commit is contained in:
Alexey Milovidov
parent
97f737b4aa
commit
9eafc13f3b
@ -1,9 +1,11 @@
|
||||
ClickHouse Testing
|
||||
# ClickHouse Testing
|
||||
|
||||
|
||||
=== Functional Tests
|
||||
## Functional Tests
|
||||
|
||||
Functional tests are the most simple and convenient to use. Each functional test sends one or multiple queries to the running ClickHouse server and compares the result with reference.
|
||||
Functional tests are the most simple and convenient to use. Most of ClickHouse features can be tested with functional tests and they are mandatory to use for every change in ClickHouse code that can be tested that way.
|
||||
|
||||
Each functional test sends one or multiple queries to the running ClickHouse server and compares the result with reference.
|
||||
|
||||
Tests are located in `dbms/src/tests/queries` directory. There are two subdirectories; `stateless` and `stateful`. Stateless tests run queries without any preloaded test data - they often create small synthetic datasets on the fly, within the test itself. Stateful tests require preloaded test data from Yandex.Metrica and not available to general public. We tend to use only `stateless` tests and avoid adding new `stateful` tests.
|
||||
|
||||
@ -22,7 +24,7 @@ If you want to use distributed queries in functional tests, you can leverage `re
|
||||
Some tests are marked with `zookeeper`, `shard` or `long` in their names. `zookeeper` is for tests that are using ZooKeeper; `shard` is for tests that requires server to listen `127.0.0.*`; `long` is for tests that run slightly longer that one second.
|
||||
|
||||
|
||||
=== Integration Tests
|
||||
## Integration Tests
|
||||
|
||||
Integration tests allow to test ClickHouse in clustered configuration and ClickHouse interaction with other servers like MySQL, Postgres, MongoDB. They are useful to emulate network splits, packet drops, etc. These tests are run under Docker and create multiple containers with various software.
|
||||
|
||||
@ -30,50 +32,54 @@ See `dbms/tests/integration/README.md` on how to run these tests.
|
||||
|
||||
Note that integration of ClickHouse with third-party drivers is not tested. Also we currently don't have integration tests with our JDBC and ODBC drivers.
|
||||
|
||||
We don't have integration tests for `Kafka` table engine that is developed by community - this is one of the most anticipated tests (otherwise there is almost no way to be confident with `Kafka` tables).
|
||||
|
||||
=== Unit Tests
|
||||
|
||||
## Unit Tests
|
||||
|
||||
Unit tests are useful when you want to test not the ClickHouse as a whole, but a single isolated library or class. You can enable or disable build of tests with `ENABLE_TESTS` CMake option. Unit tests (and other test programs) are located in `tests` subdirectories across the code. To run unit tests, type `ninja test`. Some tests use `gtest`, but some are just programs that return non-zero exit code on test failure.
|
||||
|
||||
It's not necessarily to have unit tests if the code is already covered with functional tests (and functional tests are usually much more simple to use).
|
||||
It's not necessarily to have unit tests if the code is already covered by functional tests (and functional tests are usually much more simple to use).
|
||||
|
||||
|
||||
=== Performance Tests
|
||||
## Performance Tests
|
||||
|
||||
Performance tests allow to measure and compare performance of some isolated part of ClickHouse on synthetic queries. Tests are located at `dbms/tests/performance`. Each test is represented by `.xml` file with description of test case. Tests are run with `clickhouse performance-test` tool (that is embedded in `clickhouse` binary). See `--help` for the invocation.
|
||||
Performance tests allow to measure and compare performance of some isolated part of ClickHouse on synthetic queries. Tests are located at `dbms/tests/performance`. Each test is represented by `.xml` file with description of test case. Tests are run with `clickhouse performance-test` tool (that is embedded in `clickhouse` binary). See `--help` for invocation.
|
||||
|
||||
Each test run one or miltiple queries (possibly with combinations of parameters) in a loop with some conditions for stop (like "maximum execution speed is not changing in three seconds") and measure some metrics about query performance (like "maximum execution speed"). Some tests can contain preconditions on preloaded test dataset.
|
||||
|
||||
If you want to improve performance of ClickHouse in some scenario, and if improvements can be observed on simple scenarios, it is highly recommended to write a performance test. It always makes sense to use `perf top` or other perf tools during your tests.
|
||||
If you want to improve performance of ClickHouse in some scenario, and if improvements can be observed on simple queries, it is highly recommended to write a performance test. It always makes sense to use `perf top` or other perf tools during your tests.
|
||||
|
||||
Performance tests are not run on per-commit basis. Results of performance tests are not collected and we compare them manually.
|
||||
|
||||
|
||||
=== Test tools and scripts
|
||||
## Test tools and scripts
|
||||
|
||||
Some programs in `tests` directory are not prepared tests, but are test tools. For example, for `Lexer` there is a tool `dbms/src/Parsers/tests/lexer` that just do tokenization of stdin and writes colorized result in stdout. You can use these kind of tools as a code examples and for the exploration and manual testing.
|
||||
Some programs in `tests` directory are not prepared tests, but are test tools. For example, for `Lexer` there is a tool `dbms/src/Parsers/tests/lexer` that just do tokenization of stdin and writes colorized result to stdout. You can use these kind of tools as a code examples and for exploration and manual testing.
|
||||
|
||||
You can also place pair of files `.sh` and `.reference` along with the tool to run it on some predefined input - then script result can be compared with `.reference` file. There kind of tests are not automated.
|
||||
You can also place pair of files `.sh` and `.reference` along with the tool to run it on some predefined input - then script result can be compared to `.reference` file. There kind of tests are not automated.
|
||||
|
||||
|
||||
=== Miscellanous Tests
|
||||
## Miscellanous Tests
|
||||
|
||||
There are tests for external dictionaries located at `dbms/tests/external_dictionaries` and for machine learning models in `dbms/tests/external_models`. These tests are not updated and must be transferred to integration tests.
|
||||
There are tests for external dictionaries located at `dbms/tests/external_dictionaries` and for machine learned models in `dbms/tests/external_models`. These tests are not updated and must be transferred to integration tests.
|
||||
|
||||
There is separate test for quorum inserts. This test run ClickHouse cluster on separate servers and emulate various fail cases: network split, packet drop (between ClickHouse nodes, between ClickHouse and ZooKeeper, between ClickHouse server and client, etc.), `kill -9`, `kill -STOP` and `kill -CONT` , like [Jepsen](https://aphyr.com/tags/Jepsen). Then test checks that all acknowledged inserts was written.
|
||||
There is separate test for quorum inserts. This test run ClickHouse cluster on separate servers and emulate various failure cases: network split, packet drop (between ClickHouse nodes, between ClickHouse and ZooKeeper, between ClickHouse server and client, etc.), `kill -9`, `kill -STOP` and `kill -CONT` , like [Jepsen](https://aphyr.com/tags/Jepsen). Then the test checks that all acknowledged inserts was written and all rejected inserts was not.
|
||||
|
||||
Quorum test was written by separate team before ClickHouse was open-sourced. This team no longer work with ClickHouse. Test was occasionally written in Java. For these reasons, quorum test must be rewritten and moved to integration tests.
|
||||
Quorum test was written by separate team before ClickHouse was open-sourced. This team no longer work with ClickHouse. Test was accidentially written in Java. For these reasons, quorum test must be rewritten and moved to integration tests.
|
||||
|
||||
|
||||
=== Manual Testing
|
||||
## Manual Testing
|
||||
|
||||
When you develop new feature, it is reasonable to also test it manually. You can do it with the following steps:
|
||||
When you develop a new feature, it is reasonable to also test it manually. You can do it with the following steps:
|
||||
|
||||
Build ClickHouse. Run ClickHouse from the terminal: change directory to `dbms/src/programs/clickhouse-server` and run it with `./clickhouse-server`. It will use configuration (`config.xml`, `users.xml` and files within `config.d` and `users.d` directories) from the current directory by default. To connect to ClickHouse server, run `dbms/src/programs/clickhouse-client/clickhouse-client`.
|
||||
|
||||
Note that all clickhouse tools (server, client, etc) are just symlinks for a single binary named `clickhouse`. You can find this binary at `dbms/src/programs/clickhouse`. All tools can also be invoked as `clickhouse tool` instead of `clickhouse-tool`.
|
||||
Note that all clickhouse tools (server, client, etc) are just symlinks to a single binary named `clickhouse`. You can find this binary at `dbms/src/programs/clickhouse`. All tools can also be invoked as `clickhouse tool` instead of `clickhouse-tool`.
|
||||
|
||||
Alternatively you can install ClickHouse package: either stable release from Yandex repository or you can build package for yourself with `./release` in ClickHouse sources root. Then start with `sudo service clickhouse-server start` or stop. Look for logs at `/etc/clickhouse-server/clickhouse-server.log`.
|
||||
Alternatively you can install ClickHouse package: either stable release from Yandex repository or you can build package for yourself with `./release` in ClickHouse sources root. Then start the server with `sudo service clickhouse-server start` (or stop to stop the server). Look for logs at `/etc/clickhouse-server/clickhouse-server.log`.
|
||||
|
||||
When ClickHouse is already installed in your system, you can build new `clickhouse` binary and replace existing binary:
|
||||
When ClickHouse is already installed on your system, you can build a new `clickhouse` binary and replace the existing binary:
|
||||
```
|
||||
sudo service clickhouse-server stop
|
||||
sudo cp ./clickhouse /usr/bin/
|
||||
@ -91,14 +97,14 @@ Example with gdb:
|
||||
sudo -u clickhouse gdb --args /usr/bin/clickhouse server --config-file /etc/clickhouse-server/config.xml
|
||||
```
|
||||
|
||||
If system clickhouse-server is already running and you don't want to stop it, you can change port numbers in your `config.xml` (or override them in a file in `config.d` directory), provide appropriate data path, and run it.
|
||||
If the system clickhouse-server is already running and you don't want to stop it, you can change port numbers in your `config.xml` (or override them in a file in `config.d` directory), provide appropriate data path, and run it.
|
||||
|
||||
`clickhouse` binary have almost no dependencies and works across wide range of Linux distributions. To quick and dirty test your changes on a server, you can simply `scp` your fresh built `clickhouse` binary to your server and then run it as in examples above.
|
||||
`clickhouse` binary has almost no dependencies and works across wide range of Linux distributions. To quick and dirty test your changes on a server, you can simply `scp` your fresh built `clickhouse` binary to your server and then run it as in examples above.
|
||||
|
||||
|
||||
=== Testing Environment
|
||||
## Testing Environment
|
||||
|
||||
Before publishing release as stable we deploy it on testing environment. Testing environment is a cluster that process 1/39 part of Yandex.Metrica data. We share our testing environment with Yandex.Metrica team. ClickHouse is upgraded without downtime on top of existing data. We look at first that data is processed successfully without delay from realtime, the replication continue to work and there is no visible issues for Yandex.Metrica team. First check can be done like this:
|
||||
Before publishing release as stable we deploy it on testing environment. Testing environment is a cluster that process 1/39 part of [Yandex.Metrica](https://metrica.yandex.com/) data. We share our testing environment with Yandex.Metrica team. ClickHouse is upgraded without downtime on top of existing data. We look at first that data is processed successfully without lagging from realtime, the replication continue to work and there is no issues visible to Yandex.Metrica team. First check can be done in the following way:
|
||||
|
||||
```
|
||||
SELECT hostName() AS h, any(version()), any(uptime()), max(UTCEventTime), count() FROM remote('example01-01-{1..3}t', merge, hits) WHERE EventDate >= today() - 2 GROUP BY h ORDER BY h;
|
||||
@ -107,7 +113,7 @@ SELECT hostName() AS h, any(version()), any(uptime()), max(UTCEventTime), count(
|
||||
In some cases we also deploy to testing environment of our friend teams in Yandex: Market, Cloud, etc. Also we have some hardware servers that are used for development purposes.
|
||||
|
||||
|
||||
=== Load Testing
|
||||
## Load Testing
|
||||
|
||||
After deploying to testing environment we run load testing with queries from production cluster. This is done manually.
|
||||
|
||||
@ -127,14 +133,14 @@ clickhouse benchmark --concurrency 16 < queries.tsv
|
||||
```
|
||||
(probably you also want to specify a `--user`)
|
||||
|
||||
Then leave it for night or weekend and go take a rest.
|
||||
Then leave it for a night or weekend and go take a rest.
|
||||
|
||||
You should check that `clickhouse-server` doesn't crash, memory footprint is bounded and performance not degrading over time.
|
||||
|
||||
Precise query execution timings are not recorded and not compared due to high variability of queries and environment.
|
||||
|
||||
|
||||
=== Build Tests
|
||||
## Build Tests
|
||||
|
||||
Build tests allow to check that build is not broken on various alternative configurations and on some foreign systems. Tests are located at `ci` directory. They run build from source inside Docker, Vagrant, and sometimes with `qemu-user-static` inside Docker. These tests are under development and test runs are not automated.
|
||||
|
||||
@ -147,17 +153,17 @@ Normally we release and run all tests on a single variant of ClickHouse build. B
|
||||
- build with shared linking of libraries;
|
||||
- build on AArch64 platform.
|
||||
|
||||
For example, build with system packages is a bad practice, because we cannot guarantee what exact version of packages a system will have. But this is really needed by Debian maintainers. For this reason we at least have to support this variant of build. Another example: shared linking is a common source of trouble, but it is needed for RHEL users.
|
||||
For example, build with system packages is bad practice, because we cannot guarantee what exact version of packages a system will have. But this is really needed by Debian maintainers. For this reason we at least have to support this variant of build. Another example: shared linking is a common source of trouble, but it is needed for some enthusiasts.
|
||||
|
||||
Though we cannot run all tests on all variant of builds, we want to check at least that various build variants are not broken. For this purpose we use build tests.
|
||||
|
||||
|
||||
=== Testing for protocol compatibility
|
||||
## Testing for protocol compatibility
|
||||
|
||||
When we extend ClickHouse network protocol, we test manually that old clickhouse-client works with new clickhouse-server and new clickhouse-client works with old clickhouse-server (simply by running binaries from corresponding packages).
|
||||
|
||||
|
||||
=== Help From The Compiler
|
||||
## Help From The Compiler
|
||||
|
||||
Main ClickHouse code (that is located in `dbms` directory) is built with `-Wall -Wextra -Werror` and with some additional enabled warnings. Although these options are not enabled for third-party libraries.
|
||||
|
||||
@ -166,13 +172,13 @@ Clang has even more useful warnings - you can look for them with `-Weverything`
|
||||
For production builds, gcc is used (it still generates slightly more efficient code than clang). For development, clang is usually more convenient to use. You can build on your own machine with debug mode (to save battery of your laptop), but please note that compiler is able to generate more warnings with `-O3` due to better control flow and inter-procedure analysis. When building with clang, `libc++` is used instead of `libstdc++` and when building with debug mode, debug version of `libc++` is used that allows to catch more errors at runtime.
|
||||
|
||||
|
||||
=== Sanitizers
|
||||
## Sanitizers
|
||||
|
||||
Address sanitizer.
|
||||
We run functional tests under ASan on per-commit basis.
|
||||
|
||||
Valgrind (Memcheck).
|
||||
We run functional tests under Valgrind overnight. It takes multiple hours. Currently there is one known false positive in `re2` library, see https://research.swtch.com/sparse.
|
||||
We run functional tests under Valgrind overnight. It takes multiple hours. Currently there is one known false positive in `re2` library, see [this article](https://research.swtch.com/sparse).
|
||||
|
||||
Thread sanitizer.
|
||||
We run functional tests under TSan. ClickHouse must pass all tests. Run under TSan is not automated and performed only occasionally.
|
||||
@ -181,52 +187,65 @@ Memory sanitizer.
|
||||
Currently we still don't use MSan.
|
||||
|
||||
Undefined behaviour sanitizer.
|
||||
We still don't use UBSan. The only thing to fix is unaligned placement of structs in Arena during aggregation.
|
||||
We still don't use UBSan. The only thing to fix is unaligned placement of structs in Arena during aggregation. This is totally fine, we only have to force alignment under UBSan.
|
||||
|
||||
Debug allocator.
|
||||
You can enable debug version of tcmalloc with `DEBUG_TCMALLOC` CMake option. We run tests with debug allocator on per-commit basis.
|
||||
You can enable debug version of `tcmalloc` with `DEBUG_TCMALLOC` CMake option. We run tests with debug allocator on per-commit basis.
|
||||
|
||||
You will find some additional details in `dbms/tests/instructions/sanitizers.txt`.
|
||||
|
||||
|
||||
=== Fuzzing
|
||||
## Fuzzing
|
||||
|
||||
As of July 2018 we don't use fuzzing.
|
||||
|
||||
|
||||
=== Security Audit
|
||||
## Security Audit
|
||||
|
||||
People from Yandex Cloud department do some basic overview of ClickHouse capabilities from the security standpoint.
|
||||
|
||||
|
||||
=== Static Analyzers
|
||||
## Static Analyzers
|
||||
|
||||
We use static analyzers only occasionally. We have evaluated `clang-tidy`, `Coverity`, `cppcheck`, `PVS-Studio`, `tscancode`. You will find instructions for usage in `dbms/tests/instructions/` directory. Also you can read [the article in russian](https://habr.com/company/yandex/blog/342018/).
|
||||
|
||||
If you use `CLion` as an IDE, you can use some `clang-tidy` checks out of the box.
|
||||
If you use `CLion` as an IDE, you can leverage some `clang-tidy` checks out of the box.
|
||||
|
||||
|
||||
=== Hardening
|
||||
## Hardening
|
||||
|
||||
`FORTIFY_SOURCE` is used by default. It is almost useless, but still makes sense in rare cases and we don't disable it.
|
||||
|
||||
|
||||
=== Code Style
|
||||
## Code Style
|
||||
|
||||
Code style rules are described [here](https://clickhouse.yandex/docs/en/development/style/).
|
||||
|
||||
To check for few common style violations, you can use `utils/check-style` script.
|
||||
To check for some common style violations, you can use `utils/check-style` script.
|
||||
|
||||
To force proper style of your code, you can use `clang-format`. File `.clang-format` is located at sources root. It mostly corresponds to our actual code style. But it's not recommended to apply `clang-format` to existing files because it makes formatting worse. You can use `clang-format-diff` tool that you can find in clang source repository.
|
||||
To force proper style of your code, you can use `clang-format`. File `.clang-format` is located at the sources root. It mostly corresponding with our actual code style. But it's not recommended to apply `clang-format` to existing files because it makes formatting worse. You can use `clang-format-diff` tool that you can find in clang source repository.
|
||||
|
||||
Alternatively you can try `uncrustify` tool to reformat your code. Configuration is in `uncrustify.cfg` in the sources root. It is less tested than `clang-format`.
|
||||
|
||||
|
||||
=== Test Automation
|
||||
## Metrica B2B Tests
|
||||
|
||||
Each ClickHouse release is tested with Yandex Metrica and AppMetrica engines. Testing and stable versions of ClickHouse are deployed on VMs and run with a small copy of Metrica engine that is processing fixed sample of input data. Then results of two instances of Metrica engine are compared together.
|
||||
|
||||
These tests are automated by separate team. Due to high number of moving parts, tests are fail most of the time by completely unrelated reasons, that are very difficult to figure out. Most likely these tests have negative value for us. Nevertheless these tests was proved to be useful in about one or two times out of hundreds.
|
||||
|
||||
|
||||
|
||||
|
||||
=== Test Coverage
|
||||
## Test Coverage
|
||||
|
||||
As of July 2018 we don't track test coverage.
|
||||
|
||||
|
||||
## Test Automation
|
||||
|
||||
We run tests with Travis CI (available for general public) and Jenkins (available inside Yandex).
|
||||
|
||||
In Travis CI due to limit on time and computational power we can afford only subset of functional tests that are run with limited build of ClickHouse (debug version with cut off most of libraries). In about half of runs it still fails to finish in 50 minutes timeout. The only advantage - test results are visible for all external contributors.
|
||||
|
||||
In Jenkins we run functional tests for each commit and for each pull request from trusted users; the same under ASan; we also run quorum tests, dictionary tests, Metrica B2B tests. We use Jenkins to prepare and publish releases. Worth to note that we are not happy with Jenkins at all.
|
||||
|
||||
One of our goals is to provide reliable testing infrastructure that will be available to community.
|
||||
|
||||
Loading…
Reference in New Issue
Block a user