Raúl Marín
a6c34fdbc9
Improve how queries are output in the performance dashboard
2021-11-25 11:22:47 +01:00
alesapin
c86b43c7ef
Merge pull request #31685 from ClickHouse/tavplubix-patch-5
...
Try fix OOMs with TSAN
2021-11-25 11:34:05 +03:00
alesapin
58fad90351
Merge branch 'master' into add_performance_tests
2021-11-25 10:53:10 +03:00
alesapin
cf2f019caa
Disable full run in perf tests
2021-11-24 23:05:27 +03:00
Alexey Milovidov
21f112b30e
Merge branch 'better-crosscompiling-freebsd' into hermetic-builds
2021-11-24 22:43:30 +03:00
Alexey Milovidov
92031e6b31
Fix PVS-Studio
2021-11-24 20:25:11 +03:00
tavplubix
82e4f941a4
Update Dockerfile
2021-11-24 16:00:15 +03:00
tavplubix
556ccd0bc9
Update Dockerfile
2021-11-24 12:47:44 +03:00
Alexey Milovidov
51ee8fdbb0
More easy cross-compiling for FreeBSD
2021-11-24 00:32:04 +03:00
alexey-milovidov
61454ffbcb
Merge branch 'master' into hermetic-builds
2021-11-24 00:25:16 +03:00
alesapin
42c77323e8
Merge branch 'master' into add_performance_tests
2021-11-23 22:14:32 +03:00
alexey-milovidov
341f705419
Merge pull request #30599 from ClickHouse/zlib-ng-own-cmake
...
Use our own CMakeLists for `zlib-ng`, `cassandra`, `mariadb-connector-c` and `xz`, `re2`, `sentry`, `gsasl`, `arrow`, `protobuf`.
2021-11-23 13:33:29 +03:00
alesapin
2396f73e08
Fix performance tests comparison
2021-11-23 11:34:10 +03:00
alesapin
62b2eb6694
Merge branch 'master' into add_performance_tests
2021-11-23 11:26:45 +03:00
Constantine Peresypkin
fd551ad94c
fix kerberized_hadoop image
...
current image uses non-existing centos6 repos
added direct fetch of actual rpm packages
fixed a typo in docker/images.json: duplicate entry
2021-11-22 17:32:19 +02:00
Alexey Milovidov
d604cf5573
Remove the infamous "unbundled" build
2021-11-21 13:58:26 +03:00
Azat Khuzhin
d0ecdb617e
Remove duplicated not compressed logs from CI (actions)
...
Right now we cannot remove that compression since non-github actions
check will upload non compressed files then, but once old CI will be
removed it seems to be possible to remove that manual compression.
2021-11-19 21:33:48 +03:00
alesapin
9e440988cf
Add datasets
2021-11-19 16:10:49 +03:00
alesapin
130c09f348
Fix search for old version
2021-11-18 13:38:59 +03:00
alesapin
d0ec0c5dc8
Merge branch 'master' into add_performance_tests
2021-11-18 10:17:20 +03:00
alesapin
b74f9e7047
Merge branch 'master' into add_performance_tests
2021-11-18 10:12:46 +03:00
Nikolay Degterinsky
fc9ef14a73
Merge pull request #29219 from evillique/log_lz4_streaming
...
Add LZ4 stream compression of logs
2021-11-17 21:22:57 +03:00
Nikolay Degterinsky
eafd3ddfac
Add lz4 stream compression
2021-11-16 22:30:29 +00:00
tavplubix
b1079ae309
Update run.sh
2021-11-16 20:03:50 +03:00
tavplubix
224a4c1b6b
Update run.sh
2021-11-16 17:45:37 +03:00
alesapin
7f09941c72
Add performance comparison check
2021-11-16 14:16:10 +03:00
alexey-milovidov
f4fda976ef
Merge pull request #31063 from azat/perf-spikes
...
perf: switch *_log tables to Memory engine (attempt to reduce cache misses)
2021-11-14 05:13:12 +03:00
Azat Khuzhin
48b1b7b7ec
Fix test_kafka_insert_avro by pinning avro version
...
New avro version has a code that does not allow to use BytesIO [1].
[1]: https://github.com/apache/avro/commit/e8c61ba4f0#diff-ca721cc66db80ff81dcd7b8ad1bb0e83b2e5c2de80c207ebec79f5495d38db98R318-R319
2021-11-13 12:01:13 +03:00
Azat Khuzhin
40be97c1dd
Improve fuzzer report in case of fuzzer killed
...
This will avoid Exception from fuzzer in description like in [1], that
someone may think that is related to the failure, which is obviously
don't in case it was KILLed.
[1]: https://clickhouse-test-reports.s3.yandex.net/31259/a49e10c78860a451ff9cc39df6049c88234d507a/fuzzer_ubsan/report.html#fail1
2021-11-12 01:01:10 +03:00
mergify[bot]
045bd40070
Merge branch 'master' into perf-spikes
2021-11-11 07:07:28 +00:00
alesapin
0b28474f09
Merge pull request #31244 from nikitamikhaylov/disable-fuzz-builds
...
Disable fuzzer builds in CI
2021-11-10 16:42:08 +03:00
Nikita Mikhaylov
99f78472bc
Disabled
2021-11-10 11:38:26 +00:00
Kseniia Sumarokova
5f9b35b982
Update docker_compose_postgres.yml
2021-11-10 10:36:59 +03:00
Kseniia Sumarokova
5eef7e0e45
Update docker_compose_postgres.yml
2021-11-09 15:13:01 +03:00
Azat Khuzhin
9e622b5add
perf: switch *_log tables to Memory engine (attempt to reduce cache misses)
...
trace_log/query_log from performance tests shows (for
cases when prewarm query fails with timeout, 15sec) excessive
writeTraceInfo() in trace_log and QueryProfilerRuns in query_log, but
this is not the root cause of the timeout, but consequence.
Also query_log shows that on failures the following profile events has
significantly higher values:
- PerfLocalMemoryMisses (6.3x more)
- PerfLocalMemoryReferences (7x more)
- PerfDataTLBMisses (6.9x more)
- PerfInstructionTLBMisses (6.4x more)
During looking at performance tests logs I noticed that once the prewarm
query fails other server (left/right) was merging (MergeTree) something
in *_log tables.
But, using MergeTree for *_log in performance tests is useless, since
anyway environment for performance tests uses ramdrive.
And so MergeTree merges just increase overhead.
Eventually I expect that this should decrease extra memory referencing
and so this should decrease cache/TLB misses.
CI: https://clickhouse-test-reports.s3.yandex.net/30886/c504e0c08df7a926bb479a1d297f326f5c48a32f/performance_comparison/report.html#fail1
v2: <partition_by remove="remove"/>
2021-11-04 09:26:46 +03:00
Azat Khuzhin
ff19d22305
perf: remove over *_log tables that does not exported into artifacts
2021-11-03 08:29:18 +03:00
Azat Khuzhin
837fe8554f
perf: remove zookeeper to avoid configuring DDLWorker
2021-11-03 08:29:18 +03:00
Azat Khuzhin
acef85b17d
perf: cleanup server overrides
2021-11-03 08:29:18 +03:00
Azat Khuzhin
afd960db4e
perf: remove redundant query profiler set queries
2021-11-03 08:09:43 +03:00
robot-clickhouse
4289336c27
Auto version update to [21.12.1.1] [54457]
2021-11-02 00:56:45 +03:00
Kruglov Pavel
9a1275cb10
Merge pull request #30178 from Avogar/tsv-csv
...
Refactor and improve TSV, CSV, JSONCompactEachRow, RowBinary formats. Fix bugs in formats
2021-11-02 00:38:30 +03:00
alesapin
dbfa17aabf
Add correct S3
2021-11-01 13:32:56 +03:00
avogar
4e63311bb6
Try fix perf test
2021-11-01 12:13:29 +03:00
alesapin
54e404de32
Add github to known hosts in docs release
2021-11-01 10:33:52 +03:00
alesapin
ed3d707513
Merge pull request #30839 from ClickHouse/qoega-docker-docs
...
Docs check on github actions
2021-10-31 20:29:04 +03:00
alexey-milovidov
15b9d65221
Merge pull request #30662 from azat/conf-clickhouse
...
Switch everything left from `<yandex>` to `<clickhouse>`
2021-10-31 14:56:51 +03:00
alesapin
d872eae809
Better
2021-10-29 21:43:03 +03:00
alesapin
4c645f3ef6
Add docs release
2021-10-29 21:43:03 +03:00
alesapin
1a13a741f9
Fixes in docs
2021-10-29 21:43:03 +03:00
alesapin
6c8ba21c52
Add docs check image
2021-10-29 21:43:03 +03:00
alesapin
11f71fffe1
Tabs to spaces
2021-10-29 21:43:03 +03:00
alesapin
afcadc95a7
Add docs check to documentation
2021-10-29 21:43:03 +03:00
Yatsishin Ilya
faeb4ef869
Add docker container to build docs
2021-10-29 21:43:03 +03:00
tavplubix
059c1ebf36
Merge pull request #30847 from ClickHouse/fix_30802
...
Disable check for ZooKeeper session uptime by default
2021-10-29 20:18:25 +03:00
Alexander Tokmakov
d095cfe4c1
disable check for zk session uptime by default
2021-10-29 14:45:09 +03:00
alesapin
4eca493ab8
Fix fuzzer on master
2021-10-29 13:47:29 +03:00
alesapin
11b70a285c
Merge pull request #30795 from ClickHouse/make_fuzzer_check_less_hardcoded
...
Trying to make fuzzer check less hardcoded
2021-10-29 10:21:36 +03:00
alesapin
ddac7a51e2
Merge pull request #30729 from ClickHouse/trying_actions
...
Add functional tests to github actions
2021-10-29 09:49:31 +03:00
mergify[bot]
65df198e9b
Merge branch 'master' into conf-clickhouse
2021-10-28 22:19:42 +00:00
alesapin
ffc8dbd085
Update run-fuzzer.sh
2021-10-28 23:06:04 +03:00
alesapin
313584e553
Update run-fuzzer.sh
2021-10-28 19:12:47 +03:00
mergify[bot]
a94b8902dd
Merge branch 'master' into trying_actions
2021-10-28 15:57:28 +00:00
alesapin
0651a1f0e9
Update run-fuzzer.sh
2021-10-28 17:03:40 +03:00
alesapin
542196128b
Trying to make fuzzer check less hardcoded
2021-10-28 12:29:50 +03:00
alesapin
c0b2f92123
Add stateful test
2021-10-28 12:26:59 +03:00
Azat Khuzhin
9eef2ad0c2
perf: reduce number of times the query will run
...
Before this patch:
- upstream/master and PR's *with* perf tests or pef scripts changes:
--runs=13 --max-queries=0
- PRs *without* perf changes:
--runs=7 --max-queries=20
- PRs w/ only perf tests changes:
--runs-13 --max-queries=0 <list of perf tests>
After:
- upstream/master and PR's *with* perf tests changes:
--runs=13 --max-queries=0
- PRs *without* perf changes:
--runs=7 --max-queries=10
- PRs w/ only perf tests changes:
--runs-13 --max-queries=0 <list of perf tests>
So to underline, now we will not look at perf scripts changes anymore,
and we will also decrease number of random queries to run to 10.
2021-10-27 23:20:51 +03:00
Azat Khuzhin
7957079f95
perf: disable query/memory profiling for non prewarm queries
2021-10-27 23:20:51 +03:00
Azat Khuzhin
190459db33
perf: add queries with errors during process too to the perf artifacts
...
Sometimes queries fails with timeout, and you will not get any perf
metrics for it before this patch.
Example: https://clickhouse-test-reports.s3.yandex.net/30611/5ff6c5536558821824d5fdf25a75729e5b82060d/performance_comparison/report.html#partial-queries.flat_dictionary.5
2021-10-27 23:20:51 +03:00
Azat Khuzhin
722d528218
perf: pin clickhouse-driver to 0.2.1 for tzlocal <0.3 to avoid warning
...
CI report [1]:
/usr/local/lib/python3.6/dist-packages/clickhouse_driver/columns/datetimecolumn.py:199: PytzUsageWarning: The zone attribute is specific to pytz's interface; please migrate to a new time zone provider. For more details on how to do so, see https://pytz-deprecation-shim.readthedocs.io/en/latest/migration.html
[1]: https://clickhouse-test-reports.s3.yandex.net/30626/dfc85841134aa96b4e04c401462898eb305e8657/performance_comparison/report.html#changes-in-performance.format_date_time.1
2021-10-27 23:20:51 +03:00
Azat Khuzhin
90ea2de9bb
Add unidiff into style image
2021-10-27 10:03:24 +03:00
Azat Khuzhin
b41c77f4f8
Add boto3 to style image
2021-10-27 10:03:24 +03:00
Azat Khuzhin
944437caf6
Add PyGithub into style image
2021-10-27 10:03:23 +03:00
Nikita Mikhaylov
a4100cc980
Build local in CI in fuzz mode
2021-10-26 17:45:41 +00:00
Nikita Mikhaylov
aae30a9e37
Fix fuzzer build ( #30344 )
2021-10-26 20:30:34 +03:00
Azat Khuzhin
da90df5658
docker: switch <yandex> to <clickhouse>
2021-10-26 08:53:14 +03:00
Azat Khuzhin
8f015764a7
Add metadata for *_log into tests artifacts
...
This will allow to use artifacts as-is, i.e.:
clickhouse-local --path /path/to/log/artifact
v2: convert database to Ordinary to avoid extra symlinks dependencies.
2021-10-23 20:17:01 +03:00
alesapin
9e8dfdb089
Merge pull request #28851 from ClickHouse/trying_actions
...
Trying github actions
2021-10-20 12:55:53 +03:00
alesapin
6d8853698b
Merge branch 'master' into trying_actions
2021-10-19 17:45:37 +03:00
vdimir
acf416900d
Minor fix in clickhouse/kerberized-hadoop Dockerfile
2021-10-19 16:45:03 +03:00
vdimir
1d78f1c63b
Fix ca-bundle.crt in clickhouse/kerberized-hadoop
2021-10-19 16:40:06 +03:00
vdimir
36bb4033ba
Whitespace change in kerberized_hadoop/Dockerfile
2021-10-19 12:44:55 +03:00
Alexey Milovidov
2cef9983fa
Remove recursive submodules
2021-10-18 02:29:04 +03:00
Alexey Milovidov
f02b61e30f
Merge branch 'master' into hermetic-builds
2021-10-17 07:02:37 +03:00
mergify[bot]
f29ea07b1c
Merge branch 'master' into clickhouse-test-http-interface
2021-10-13 16:46:26 +00:00
tavplubix
a16ce6b54b
Merge pull request #30030 from ClickHouse/tavplubix-patch-3
...
Attach trace_log to stateless tests report as is
2021-10-13 13:21:39 +03:00
Alexey Milovidov
b420bc19fb
Merge branch 'master' into hermetic-builds
2021-10-13 01:08:23 +03:00
Dmitry Novik
fa9cdd5c5f
Use original whitespaces in test_results.tsv
2021-10-12 23:31:31 +03:00
Dmitry Novik
0507391010
Do not stop CI tests processing if clickhouse-test return 1
2021-10-12 23:30:07 +03:00
Dmitry Novik
a30573fc0a
Add FAIL message to test_results.tsv
2021-10-12 23:29:15 +03:00
Azat Khuzhin
071a5af96e
Revert "docker: add pandas/clickhouse_driver into test images"
...
This reverts commit e07a6f3fc0
2021-10-12 21:06:26 +03:00
alexey-milovidov
9a1f930b2f
Merge pull request #30010 from ClickHouse/ppc64le
...
Add cross-build for PowerPC 64le
2021-10-12 16:17:41 +03:00
tavplubix
f3c9f4be6c
Update run.sh
2021-10-12 14:38:41 +03:00
alesapin
2b3a0c892d
Merge pull request #29992 from ClickHouse/more_timeouts_in_scripts
...
More timeouts in test scripts
2021-10-12 10:14:21 +03:00
Alexey Milovidov
bdd06279a4
Fix PVS Studio
2021-10-12 05:43:53 +03:00
Alexey Milovidov
d0f7a10c6a
Fix fasttest
2021-10-12 03:34:07 +03:00
Alexey Milovidov
75547e64a5
Add to packager
2021-10-12 02:10:58 +03:00
alesapin
3ae960e04b
Review fixes
2021-10-11 16:40:12 +03:00
alesapin
9ad919d91a
More timeouts in stress test
2021-10-11 13:25:54 +03:00
Alexey Milovidov
0dbe7c4036
Remove old toolchain
2021-10-11 07:02:19 +03:00
alexey-milovidov
268c155b7d
Merge pull request #29856 from azat/clickhouse-test-python-client
...
Rewrite clickhouse-test to use python clickhouse_driver
2021-10-09 17:58:40 +03:00
alexey-milovidov
f6a0983068
Merge pull request #29781 from azat/fuzzer-server-pid-v2
...
Simplify obtaining of server pid in fuzzer tests
2021-10-08 00:35:25 +03:00
Azat Khuzhin
e07a6f3fc0
docker: add pandas/clickhouse_driver into test images
2021-10-08 00:09:37 +03:00
mergify[bot]
6eec0d76a7
Merge branch 'master' into update-minio
2021-10-06 05:45:18 +00:00
Azat Khuzhin
aeb670dbe0
Simplify obtaining of server pid in fuzzer tests
2021-10-05 21:05:49 +03:00
Nikita Mikhaylov
a880cce0de
better
2021-10-05 13:26:29 +00:00
Kseniia Sumarokova
ead99b5911
Update docker_compose_minio.yml
2021-10-05 11:39:08 +03:00
kssenii
4962677a8b
Update minio
2021-10-05 08:18:16 +00:00
Nikita Mikhaylov
731d6c5755
Merge pull request #29513 from azat/fuzzer-fix-server-pid
...
Fix server pid (hence exit code and attaching with gdb) in fuzzer tests
2021-10-04 19:59:24 +03:00
Azat Khuzhin
95d8133aaa
Cleanup unbundled image
...
No need to install specific boost packages, when all boost is installed.
2021-10-04 00:58:29 +03:00
Azat Khuzhin
74a7a0603d
test/fuzzer: print stacktrace
...
To get the stacktrace for errors w/o it [1]:
SET max_parser_depth = 4000
Got boring AST
Code: 306. DB::Exception: Maximum parse depth (1000) exceeded. Consider rising max_parser_depth parameter. (TOO_DEEP_RECURSION)
[1]: https://clickhouse-test-reports.s3.yandex.net/29513/769bfbe71f22c392458eced655080a0bbd442e02/fuzzer_ubsan/fuzzer.log
2021-10-02 12:49:44 +03:00
Azat Khuzhin
769bfbe71f
Fix server pid in fuzzer tests
...
Previously it was the pid of the subshell 40 while it should be the pid
of the clickhouse-server 39:
Here we see that the server pid is 39:
2021-09-28 11:02:34 + pgrep -f clickhouse-server
2021-09-28 11:02:34 39
Here we see that the 40 is the pid of subshell:
2021-09-28 11:02:45 ch/docker/test/fuzzer/run-fuzzer.sh: line 90: 39 Killed clickhouse-server --config-file db/config.xml -- --path db 2>&1
2021-09-28 11:02:45 40 Done | tail -100000 > server.log
And here we see that server_pid variable is 40:
2021-09-28 11:02:45 + server_exit_code=0
2021-09-28 11:02:45 + wait 40
v2: wait in background to call wait in foreground and ensure that the
process is alive, since w/o job control this is the only way to obtain
the exit code
2021-10-01 21:05:30 +03:00
alesapin
53fce1411c
Merge pull request #29597 from ClickHouse/fix-docker-image
...
Fix docker/test/util declaration
2021-10-01 13:43:17 +03:00
Alexey Milovidov
4271f8c738
Merge remote-tracking branch 'origin/master' into yandex-to-clickhouse-in-configs
2021-10-01 01:33:58 +03:00
Dmitry Novik
ad3c3f72e4
Fix docker/test/util declaration
2021-10-01 01:04:06 +03:00
Alexey Milovidov
2ec75b824b
Revert "repo.clickhouse.tech is not 100% ready"
...
This reverts commit 8616ba62e4
2021-09-29 02:43:20 +03:00
Dmitry Novik
29587c5013
Pass all parameters to process_functional_tests_result.py
2021-09-28 20:38:41 +03:00
Dmitry Novik
6b12869f6e
Fix path to executable
2021-09-28 15:11:18 +03:00
Dmitry Novik
4ce12a7685
Fixup
2021-09-28 11:50:28 +03:00
alesapin
8e25144bbb
Merge branch 'master' into trying_actions
2021-09-28 11:08:13 +03:00
Dmitry Novik
7755b3688f
Fix docker images build
2021-09-28 01:07:53 +03:00
Dmitry Novik
5418ea83fa
Fix clickhouse/test-util Dockerfile
2021-09-27 17:38:25 +03:00
Dmitry Novik
d9ce2c2d99
Update images.json
2021-09-27 15:59:20 +03:00
Dmitry Novik
4e6c945fdf
Rename fasttest output file
2021-09-27 13:09:57 +03:00
mergify[bot]
dbb924171d
Merge branch 'master' into fasttest-tsv
2021-09-27 07:48:49 +00:00
mergify[bot]
64669a28ce
Merge branch 'master' into yandex-to-clickhouse-in-configs
2021-09-26 22:59:23 +00:00
Alexey Milovidov
dc7eb5c1ae
Fix ugly typo
2021-09-26 03:29:32 +03:00
Alexey Milovidov
fb7ce28198
Merge branch 'master' into yandex-to-clickhouse-in-configs
2021-09-26 02:10:11 +03:00
Alexey Milovidov
90e74aff70
Fix performance test
2021-09-26 02:08:46 +03:00
Dmitry Novik
98df97f5d9
Fix docker image
2021-09-24 13:48:28 +03:00
Dmitry Novik
77dfa44256
Generate test_results.tsv for fasttest
2021-09-24 12:16:08 +03:00
Alexey Milovidov
8616ba62e4
repo.clickhouse.tech is not 100% ready
2021-09-24 04:41:59 +03:00
mergify[bot]
88b7680eeb
Merge branch 'master' into trying_actions
2021-09-23 15:48:57 +00:00
Alexey Milovidov
e9e77b4403
.tech -> .com
2021-09-22 03:22:57 +03:00
alexey-milovidov
0880ae2883
Merge branch 'master' into yandex-to-clickhouse-in-configs
2021-09-21 20:17:28 +03:00
tavplubix
8adfb9b593
Stop ThreadFuzzer before hung check ( #29167 )
...
* stop ThreadFuzzer before hung check
* fix
* fix
2021-09-20 17:23:10 +03:00
Alexey Milovidov
147ef79670
Change website URL, part 1
2021-09-20 01:56:28 +03:00
Alexey Milovidov
bdc99b6c91
Change Alpine Dockerfile
2021-09-20 01:53:52 +03:00
Alexey Milovidov
ab44e0addb
Change Docker org name
2021-09-20 01:52:21 +03:00
Alexey Milovidov
510540c7d7
Change Doker Hub URL
2021-09-20 01:45:18 +03:00
Alexey Milovidov
e513a5db32
Change <yandex> to <clickhouse> in configs
2021-09-20 01:38:53 +03:00
alesapin
00d9a62d6c
Missclick
2021-09-17 14:17:50 +03:00
alesapin
5d35ba7d93
One more time
2021-09-17 13:41:29 +03:00
alesapin
68c7f8638a
update docker image
2021-09-17 13:34:45 +03:00
alesapin
71dcf03a0f
Create changed image
2021-09-15 20:35:15 +03:00
Nikita Mikhaylov
4b3c0a95d1
Merge pull request #28046 from syominsergey/fix-build-clang-13-2
...
[WIP] Improve support for build with clang-13
2021-09-13 12:11:38 +03:00
Vitaly Baranov
512786346c
Merge pull request #28909 from vitlibar/convert-skip-list-into-first-line-comments
...
Convert skip_list .json into first line comments
2021-09-13 03:37:39 +03:00
Sergei Semin
d3b3e8607c
fix tabs
2021-09-12 19:51:47 +03:00
Sergei Semin
12e24a23d3
Merge branch 'master' into fix-build-clang-13-2
2021-09-12 19:30:55 +03:00
Vitaly Baranov
daa5979347
Remove unnecessary files.
2021-09-12 17:15:28 +03:00
Vitaly Baranov
bbb192ee85
Convert skip_list.json into first line comments.
2021-09-12 17:15:23 +03:00
