mirror of
https://github.com/ClickHouse/ClickHouse.git
synced 2024-11-21 15:12:02 +00:00
some updates
This commit is contained in:
Alexander Kuzmenkov
parent
f6a80dec3c
commit
0349b75cb4
124
CI.md
124
CI.md
@ -1,124 +0,0 @@
|
||||
## A small note about ClickHouse's CI system.
|
||||
### CI's checks overview
|
||||
|
||||
0. Merge with master
|
||||
- Runs only on PRs.
|
||||
- Checks if the PR can be successfully merged with the master branch.
|
||||
- Not present in the checks list if there are no conflicts.
|
||||
- Status if the conflicts are present: `Cannot fetch mergecommit`.
|
||||
1. [Docs check](#-docs-check)
|
||||
- Tries to build the ClickHouse's documentation website.
|
||||
- Checks the inner cross-links status.
|
||||
2. Description check
|
||||
- Run only on PRs.
|
||||
- Check the conformance of the PR's description with the [PULL_REQUEST_TEMPLATE.md](https://github.com/yandex/ClickHouse/blob/master/.github/PULL_REQUEST_TEMPLATE.md).
|
||||
3. Push to dockerhub
|
||||
- Checks differences between the ClickHouse's current DockerHub image.
|
||||
- Status on success "nothing to update" or "successfully updated".
|
||||
4. Marker check
|
||||
- Checks that all CI's check have been initiated.
|
||||
- Status on success: `All checks were started`.
|
||||
5. [Style check](#-style-check)
|
||||
- Checks the code style according to `utils/check-style/check-style` binary.
|
||||
- Status on success: `Style check passed`.
|
||||
6. [PVS check](#-pvs-check)
|
||||
- Static analysis tool, checks code with PVS-studio.
|
||||
7. [Fast test](#-fast-test)
|
||||
- First executable check.
|
||||
- Builds a subset of ClickHouse's code (without embedded compiler, ZooKeeper, etc.).
|
||||
- Runs a subset of tests (functional, stateless, stateful, and integration ones).
|
||||
- Blocks the start of Build check on errors.
|
||||
- Intended to detect some errors and limit the CI run time on broken commits.
|
||||
- Status on success: `fail: 0, passed: 1944, skipped: 229` (may differ).
|
||||
8. [Build check](#-build-check)
|
||||
- Builds ClickHouse in various configurations for use in further steps.
|
||||
- Uses compilers `gcc-9` and `clang-10`.
|
||||
- Uses build modes `Debug` and `RelWithDebInfo`.
|
||||
- Uses sanitizers: `undefined` (UBSan), `address` (Asan), `memory` (MSan), and `thread` (TSan).
|
||||
- Status on success: `20/20 builds are OK` (may vary).
|
||||
9. Special build check
|
||||
- Checks the code with `clang-tidy`.
|
||||
10. Compatibility check
|
||||
- Checks that `clickhouse` binary runs on distributions with old libc versions.
|
||||
11. AST fuzzer
|
||||
12. Functional stateful tests
|
||||
13. Functional stateless tests
|
||||
14. Integration tests
|
||||
15. Stress test
|
||||
16. Split build smoke test
|
||||
17. Testflows check
|
||||
18. Performance tests.
|
||||
- Intended to measure the queries' relative and absolute performance changes.
|
||||
- The longest check: takes about 5 hours to run.
|
||||
|
||||
### Docs check
|
||||
- [Status page example](https://clickhouse-test-reports.s3.yandex.net/12550/eabcc293eb02214caa6826b7c15f101643f67a6b/docs_check.html)
|
||||
- `docs_output.txt` contains the building log. [Successful result example](https://clickhouse-test-reports.s3.yandex.net/12550/eabcc293eb02214caa6826b7c15f101643f67a6b/docs_check/docs_output.txt)
|
||||
### Style check
|
||||
- [Status page example](https://clickhouse-test-reports.s3.yandex.net/12550/659c78c7abb56141723af6a81bfae39335aa8cb2/style_check.html)
|
||||
- `output.txt` contains the check resulting errors (invalid tabulation etc), blank page means no errors. [Successful result example](https://clickhouse-test-reports.s3.yandex.net/12550/659c78c7abb56141723af6a81bfae39335aa8cb2/style_check/output.txt).
|
||||
|
||||
### PVS check
|
||||
- [Status page example](https://clickhouse-test-reports.s3.yandex.net/12550/67d716b5cc3987801996c31a67b31bf141bc3486/pvs_check.html)
|
||||
- `test_run.txt.out.log` contains the building and analyzing log file. It includes only parsing or not-found errors.
|
||||
- `HTML report` contains the analysis results. For its description visit PVS's [official site](https://www.viva64.com/en/m/0036/#ID14E9A2B2CD).
|
||||
|
||||
### Fast test
|
||||
[Status page example](https://clickhouse-test-reports.s3.yandex.net/12550/67d716b5cc3987801996c31a67b31bf141bc3486/fast_test.html)
|
||||
|
||||
#### Status page files
|
||||
- `runlog.out.log` is the general log that includes all other logs.
|
||||
- `test_log.txt`
|
||||
- `submodule_log.txt` contains the messages about cloning and checkouting needed submodules.
|
||||
- `stderr.log`
|
||||
- `stdout.log`
|
||||
- `clickhouse-server.log`
|
||||
- `clone_log.txt`
|
||||
- `install_log.txt`
|
||||
- `clickhouse-server.err.log`
|
||||
- `build_log.txt`
|
||||
- `cmake_log.txt` contains messages about the C/C++ and Linux flags check.
|
||||
|
||||
#### Status page columns:
|
||||
|
||||
- *Test name* contains the name of the test (without the path e.g. all types of tests will be stripped to the name).
|
||||
- *Test status* -- one of _Skipped_, _Success_, or _Fail_.
|
||||
- *Test time, sec.* -- empty on this test.
|
||||
|
||||
### Build check
|
||||
|
||||
[Status page example](https://clickhouse-builds.s3.yandex.net/12550/67d716b5cc3987801996c31a67b31bf141bc3486/clickhouse_build_check/report.html).
|
||||
|
||||
#### Status page columns:
|
||||
|
||||
- **Compiler**: `gcc-9` or `clang-10` (or `clang-10-xx` for other architectures e.g. `clang-10-freebsd`).
|
||||
- **Build type**: `Debug` or `RelWithDebInfo` (cmake).
|
||||
- **Sanitizer**: `none` (without sanitizers), `address` (ASan), `memory` (MSan), `undefined` (UBSan), or `thread` (TSan).
|
||||
- **Bundled**: ???
|
||||
- **Splitted** ???
|
||||
- **Status**: `success` or `fail`
|
||||
- **Build log**: link to the building and files copying log, useful when build failed.
|
||||
- **Build time**.
|
||||
- **Artifacts**: build result files (with `XXX` being the server version e.g. `20.8.1.4344`).
|
||||
- `clickhouse-client_XXX_all.deb`
|
||||
- `clickhouse-common-static-dbg_XXX[+asan, +msan, +ubsan, +tsan]_amd64.deb`
|
||||
- `clickhouse-common-staticXXX_amd64.deb`
|
||||
- `clickhouse-server_XXX_all.deb`
|
||||
- `clickhouse-test_XXX_all.deb`
|
||||
- `clickhouse_XXX_amd64.buildinfo`
|
||||
- `clickhouse_XXX_amd64.changes`
|
||||
- `clickhouse`: Main built binary.
|
||||
- `clickhouse-odbc-bridge`
|
||||
- `unit_tests_dbms`: GoogleTest binary with ClickHouse unit tests.
|
||||
- `shared_build.tgz`: build with shared libraries.
|
||||
- `performance.tgz`: Special ??? build for performance tests.
|
||||
|
||||
## Where the tests are run
|
||||
|
||||
Intel Gold with 100500ZB RAM.
|
||||
|
||||
## QA
|
||||
|
||||
> What is a `Task (private network)` item on status pages?
|
||||
|
||||
It's a link to the Yandex's internal job system. Yandex guys can see the check's start time and its more verbose status.
|
||||
209
docs/en/development/CI.md
Normal file
209
docs/en/development/CI.md
Normal file
@ -0,0 +1,209 @@
|
||||
## CI checks
|
||||
|
||||
When you submit a pull request, some automated checks are ran for your code by
|
||||
the ClickHouse [continuous integration (CI) system](tests.md#test-automation).
|
||||
This happens after a repository maintainer (someone from ClickHouse team) has
|
||||
screened your code and added the `can be tested` label to your pull request.
|
||||
The results of the checks are listed on the GitHub pull request page as
|
||||
described in the [GitHub checks
|
||||
documentation](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/about-status-checks).
|
||||
If a check is failing, you might be required to fix it. This page gives an
|
||||
overview of checks you may encounter, and what you can do to fix them.
|
||||
|
||||
If it looks like the check failure is not related to your changes, it may be
|
||||
some transient failure or an infrastructure problem. Push an empty commit to
|
||||
the pull request to restart the CI checks:
|
||||
```
|
||||
git reset
|
||||
git commit --allow-empty
|
||||
git push
|
||||
```
|
||||
|
||||
If you are not sure what to do, ask a maintainer for help.
|
||||
|
||||
|
||||
### Merge with master
|
||||
|
||||
Verifies that the PR can be merged to master. If not, it will fail with the
|
||||
message 'Cannot fetch mergecommit'. To fix this check, resolve the conflict as
|
||||
described in the [GitHub
|
||||
documentation](https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/resolving-a-merge-conflict-on-github),
|
||||
or merge the `master` branch to your pull request branch using git.
|
||||
|
||||
|
||||
### Docs check
|
||||
|
||||
Tries to build the ClickHouse documentation website. It can fail if you changed
|
||||
something in the documentation. Most probable reason is that some cross-link in
|
||||
the documentation is wrong. See the check report and look for `ERROR` messages.
|
||||
|
||||
#### Report details
|
||||
|
||||
- [Status page example](https://clickhouse-test-reports.s3.yandex.net/12550/eabcc293eb02214caa6826b7c15f101643f67a6b/docs_check.html)
|
||||
- `docs_output.txt` contains the building log. [Successful result example](https://clickhouse-test-reports.s3.yandex.net/12550/eabcc293eb02214caa6826b7c15f101643f67a6b/docs_check/docs_output.txt)
|
||||
|
||||
|
||||
### Description check
|
||||
|
||||
Check that the description of your pull request conforms to the template
|
||||
[PULL_REQUEST_TEMPLATE.md](https://github.com/ClickHouse/ClickHouse/blob/master/.github/PULL_REQUEST_TEMPLATE.md).
|
||||
You have to specify a changelog category for your change (e.g., Bug Fix), and
|
||||
write a user-readable message describing the change for [CHANGELOG.md](../whats-new/changelog)
|
||||
|
||||
|
||||
### Push to dockerhub
|
||||
|
||||
Builds docker images used for build and tests.
|
||||
|
||||
|
||||
### Marker check
|
||||
|
||||
This check means that the CI system started to process the pull request. When it has 'pending' status, it means that not all checks have been started yet. After all checks have been started, it changes status to 'success'.
|
||||
|
||||
|
||||
### Style check
|
||||
|
||||
Performs some simple regex-based checks of code style, using the `utils/check-style/check-style` binary.
|
||||
If it fails, fix the style errors following the [code style guide](style.md).
|
||||
|
||||
#### Report details
|
||||
- [Status page example](https://clickhouse-test-reports.s3.yandex.net/12550/659c78c7abb56141723af6a81bfae39335aa8cb2/style_check.html)
|
||||
- `output.txt` contains the check resulting errors (invalid tabulation etc), blank page means no errors. [Successful result example](https://clickhouse-test-reports.s3.yandex.net/12550/659c78c7abb56141723af6a81bfae39335aa8cb2/style_check/output.txt).
|
||||
|
||||
|
||||
### PVS check
|
||||
Check the code with [PVS-studio](https://www.viva64.com/en/pvs-studio/), a static analysis tool. Look at the report to see the exact errors. Fix them if you can, if not -- ask a ClickHouse maintainer for help.
|
||||
|
||||
#### Report details
|
||||
- [Status page example](https://clickhouse-test-reports.s3.yandex.net/12550/67d716b5cc3987801996c31a67b31bf141bc3486/pvs_check.html)
|
||||
- `test_run.txt.out.log` contains the building and analyzing log file. It includes only parsing or not-found errors.
|
||||
- `HTML report` contains the analysis results. For its description visit PVS's [official site](https://www.viva64.com/en/m/0036/#ID14E9A2B2CD).
|
||||
|
||||
|
||||
### Fast test
|
||||
Normally this is the first check that is ran for a PR. It builds ClickHouse and
|
||||
runs most of [stateless functional tests](tests.md#functional-tests), omitting
|
||||
some. If it fails, further checks are not started until it is fixed. Look at
|
||||
the report to see which tests fail, then reproduce the failure locally as
|
||||
described [here](tests.md#functional-test-locally).
|
||||
|
||||
#### Report details
|
||||
[Status page example](https://clickhouse-test-reports.s3.yandex.net/12550/67d716b5cc3987801996c31a67b31bf141bc3486/fast_test.html)
|
||||
|
||||
##### Status page files
|
||||
- `runlog.out.log` is the general log that includes all other logs.
|
||||
- `test_log.txt`
|
||||
- `submodule_log.txt` contains the messages about cloning and checkouting needed submodules.
|
||||
- `stderr.log`
|
||||
- `stdout.log`
|
||||
- `clickhouse-server.log`
|
||||
- `clone_log.txt`
|
||||
- `install_log.txt`
|
||||
- `clickhouse-server.err.log`
|
||||
- `build_log.txt`
|
||||
- `cmake_log.txt` contains messages about the C/C++ and Linux flags check.
|
||||
|
||||
##### Status page columns:
|
||||
|
||||
- *Test name* contains the name of the test (without the path e.g. all types of tests will be stripped to the name).
|
||||
- *Test status* -- one of _Skipped_, _Success_, or _Fail_.
|
||||
- *Test time, sec.* -- empty on this test.
|
||||
|
||||
|
||||
### Build check {#build-check}
|
||||
|
||||
Builds ClickHouse in various configurations for use in further steps. You have to fix the builds that fail. Build logs often has enough information to fix the error, but you might have to reproduce the failure locally. The `cmake` options can be found in the build log, grepping for `cmake`. Use these options and follow the [general build process](build.md).
|
||||
|
||||
#### Report details
|
||||
|
||||
[Status page example](https://clickhouse-builds.s3.yandex.net/12550/67d716b5cc3987801996c31a67b31bf141bc3486/clickhouse_build_check/report.html).
|
||||
|
||||
- **Compiler**: `gcc-9` or `clang-10` (or `clang-10-xx` for other architectures e.g. `clang-10-freebsd`).
|
||||
- **Build type**: `Debug` or `RelWithDebInfo` (cmake).
|
||||
- **Sanitizer**: `none` (without sanitizers), `address` (ASan), `memory` (MSan), `undefined` (UBSan), or `thread` (TSan).
|
||||
- **Bundled**: `bundled` build uses system libraries, and `unbundled` build uses libraries from `contrib` folder.
|
||||
- **Splitted** `splitted` is a [split build](build.md#split-build)
|
||||
- **Status**: `success` or `fail`
|
||||
- **Build log**: link to the building and files copying log, useful when build failed.
|
||||
- **Build time**.
|
||||
- **Artifacts**: build result files (with `XXX` being the server version e.g. `20.8.1.4344`).
|
||||
- `clickhouse-client_XXX_all.deb`
|
||||
- `clickhouse-common-static-dbg_XXX[+asan, +msan, +ubsan, +tsan]_amd64.deb`
|
||||
- `clickhouse-common-staticXXX_amd64.deb`
|
||||
- `clickhouse-server_XXX_all.deb`
|
||||
- `clickhouse-test_XXX_all.deb`
|
||||
- `clickhouse_XXX_amd64.buildinfo`
|
||||
- `clickhouse_XXX_amd64.changes`
|
||||
- `clickhouse`: Main built binary.
|
||||
- `clickhouse-odbc-bridge`
|
||||
- `unit_tests_dbms`: GoogleTest binary with ClickHouse unit tests.
|
||||
- `shared_build.tgz`: build with shared libraries.
|
||||
- `performance.tgz`: Special package for performance tests.
|
||||
|
||||
### Special build check
|
||||
Performs static analysis and code style checks using `clang-tidy`. The report is similar to the [build check](#build-check). Fix the errors found in the build log.
|
||||
|
||||
|
||||
### Functional stateless tests
|
||||
Runs [stateless functional tests](tests.md#functional-tests) for ClickHouse
|
||||
binaries built in various configurations -- release, debug, with sanitizers,
|
||||
etc. Look at the report to see which tests fail, then reproduce the failure
|
||||
locally as described [here](tests.md#functional-test-locally). Note that you
|
||||
have to use the correct build configuration to reproduce -- a test might fail
|
||||
under AddressSanitizer but pass in Debug. Download the binary from [CI build
|
||||
checks page](build.md#you-dont-have-to-build-clickhouse), or build it locally.
|
||||
|
||||
|
||||
### Functional stateful tests
|
||||
Runs [stateful functional tests](tests.md#functional-tests). Treat them in the same way as the functional stateless tests. The difference is that they require `hits` and `visits` tables from the [Yandex.Metrica dataset](../getting-started/example-datasets/metrica.md) to run.
|
||||
|
||||
|
||||
### Integration tests
|
||||
Runs [integration tests](tests.md#integration-tests).
|
||||
|
||||
|
||||
### Testflows check
|
||||
Runs some tests using Testflows test system. See [here](https://github.com/ClickHouse/ClickHouse/tree/master/tests/testflows#running-tests-locally) how to run them locally.
|
||||
|
||||
|
||||
### Stress test
|
||||
Runs stateless functional tests concurrently from several clients to detect
|
||||
concurrency-related errors. If it fails:
|
||||
|
||||
* Fix all other test failures first;
|
||||
* Look at the report to find the server logs and check them for possible causes
|
||||
of error.
|
||||
|
||||
|
||||
### Split build smoke test
|
||||
|
||||
Checks that the server build in [split build](build.md#split-build)
|
||||
configuration can start and run simple queries. If it fails:
|
||||
|
||||
* Fix other test errors first;
|
||||
* Build the server in [split build](build.md#split-build) configuration
|
||||
locally and check whether it can start and run `select 1`.
|
||||
|
||||
|
||||
### Compatibility check
|
||||
Checks that `clickhouse` binary runs on distributions with old libc versions. If it fails, ask a maintainer for help.
|
||||
|
||||
|
||||
### AST fuzzer
|
||||
Runs randomly generated queries to catch program errors. If it fails, ask a maintainer for help.
|
||||
|
||||
|
||||
### Performance tests.
|
||||
Measure changes in query performance. This is the longest check that takes just below 6 hours to run. The performance test report is described in detail [here](https://github.com/ClickHouse/ClickHouse/tree/master/docker/test/performance-comparison#how-to-read-the-report).
|
||||
|
||||
|
||||
|
||||
## QA
|
||||
|
||||
> What is a `Task (private network)` item on status pages?
|
||||
|
||||
It's a link to the Yandex's internal job system. Yandex employees can see the check's start time and its more verbose status.
|
||||
|
||||
> Where the tests are run
|
||||
|
||||
Somewhere on Yandex internal infrastructure.
|
||||
@ -154,4 +154,13 @@ They are built for stable, prestable and testing releases as long as for every c
|
||||
|
||||
To find the freshest build from `master`, go to [commits page](https://github.com/ClickHouse/ClickHouse/commits/master), click on the first green checkmark or red cross near commit, and click to the “Details” link right after “ClickHouse Build Check”.
|
||||
|
||||
## Split build configuration {#split-build}
|
||||
|
||||
Normally ClickHouse is statically linked into a single static `clickhouse` binary with minimal dependencies. This is convenient for distribution, but it means that on every change the entire binary is linked again, which is slow and may be inconvenient for development. There is an alternative configuration which creates dynamically loaded shared libraries instead, allowing faster incremental builds. To use it, add the following flags to your `cmake` invocation:
|
||||
```
|
||||
-DUSE_STATIC_LIBRARIES=0 -DSPLIT_SHARED_LIBRARIES=1 -DCLICKHOUSE_SPLIT_BINARY=1
|
||||
```
|
||||
|
||||
Note that in this configuration there is no single `clickhouse` binary, and you have to run `clickhouse-server`, `clickhouse-client` etc.
|
||||
|
||||
[Original article](https://clickhouse.tech/docs/en/development/build/) <!--hide-->
|
||||
|
||||
@ -27,6 +27,18 @@ If you want to use distributed queries in functional tests, you can leverage `re
|
||||
|
||||
Some tests are marked with `zookeeper`, `shard` or `long` in their names. `zookeeper` is for tests that are using ZooKeeper. `shard` is for tests that requires server to listen `127.0.0.*`; `distributed` or `global` have the same meaning. `long` is for tests that run slightly longer that one second. You can disable these groups of tests using `--no-zookeeper`, `--no-shard` and `--no-long` options, respectively.
|
||||
|
||||
### Running a particular test locally {#functional-test-locally}
|
||||
|
||||
Start the ClickHouse server locally, listening on the default port (9000). To
|
||||
run, for example, the test `01428_hash_set_nan_key`, change to the repository
|
||||
folder and run the following command:
|
||||
|
||||
```
|
||||
PATH=$PATH:<path to clickhouse-client> tests/clickhouse-test 01428_hash_set_nan_key
|
||||
```
|
||||
|
||||
For more options, see `tests/clickhouse-test --help`.
|
||||
|
||||
## Known Bugs {#known-bugs}
|
||||
|
||||
If we know some bugs that can be easily reproduced by functional tests, we place prepared functional tests in `tests/queries/bugs` directory. These tests will be moved to `tests/queries/0_stateless` when bugs are fixed.
|
||||
|
||||
@ -738,19 +738,34 @@ if __name__ == '__main__':
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.queries is None and os.path.isdir('queries'):
|
||||
if args.queries and not os.path.isdir(args.queries):
|
||||
print("Cannot acces the specified directory with queries (" + args.queries + ")", file=sys.stderr)
|
||||
exit(1)
|
||||
|
||||
# Autodetect the directory with queries if not specified
|
||||
if args.queries is None:
|
||||
args.queries = 'queries'
|
||||
elif args.queries is None:
|
||||
if (os.path.isdir('/usr/local/share/clickhouse-test/queries')):
|
||||
args.queries = '/usr/local/share/clickhouse-test/queries'
|
||||
if (args.queries is None and os.path.isdir('/usr/share/clickhouse-test/queries')):
|
||||
args.queries = '/usr/share/clickhouse-test/queries'
|
||||
|
||||
if not os.path.isdir(args.queries):
|
||||
# If we're running from the repo
|
||||
args.queries = os.path.join(os.path.dirname(os.path.abspath( __file__ )), 'queries')
|
||||
|
||||
if not os.path.isdir(args.queries):
|
||||
# Next we're going to try some system directories, don't write 'stdout' files into them.
|
||||
if args.tmp is None:
|
||||
args.tmp = '/tmp/clickhouse-test'
|
||||
if args.queries is None:
|
||||
|
||||
args.queries = '/usr/local/share/clickhouse-test/queries'
|
||||
|
||||
if not os.path.isdir(args.queries):
|
||||
args.queries = '/usr/share/clickhouse-test/queries'
|
||||
|
||||
if not os.path.isdir(args.queries):
|
||||
print("Failed to detect path to the queries directory. Please specify it with '--queries' option.", file=sys.stderr)
|
||||
exit(1)
|
||||
|
||||
print("Using queries from '" + args.queries + "' directory")
|
||||
|
||||
if args.skip_list_path is None:
|
||||
args.skip_list_path = os.path.join(args.queries, 'skip_list.json')
|
||||
|
||||
|
||||
Loading…
Reference in New Issue
Block a user