thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-11-20 14:42:02 +00:00
Code Issues Packages Projects Releases Wiki Activity
85,843 Commits 9,150 Branches 1,778 Tags 2 GiB
58ee917e94
Commit Graph

2887 Commits

Author SHA1 Message Date
kssenii
b6c1d1e72e Use latest version for tests 2021-12-08 21:09:11 +00:00
avogar
51bb37715f Fix fuzzer 2021-12-08 21:29:50 +03:00
Nikita Mikhaylov
d3d4524390 Merge branch 'master' of github.com:ClickHouse/ClickHouse into get_fuzz_data 2021-12-07 10:32:03 +00:00
tavplubix
523e23cfcd
Merge pull request #32164 from azat/stress-ignore-mem-limit 
Fix stress tests
 2021-12-07 13:10:17 +03:00
Nikita Mikhaylov
80a146816c More comments and style 2021-12-06 21:34:52 +00:00
Nikita Mikhaylov
fa22fdc097 Merge branch 'master' of github.com:ClickHouse/ClickHouse into get_fuzz_data 2021-12-06 14:30:49 +00:00
Azat Khuzhin
f8bf3b1993 stress: use max_memory_usage_for_user as a soft limit to avoid memory limit exceeded 
By using max_memory_usage_for_user as a soft limit, and after
max_server_memory_usage as a hard, we can allow normal overcommit, using
max_memory_usage_for_user=0 instead of relying on max_untracked_memory.
 2021-12-06 09:05:34 +03:00
Azat Khuzhin
f9a95fce13 stress: allow memory overcommit for SELECT 1 query 2021-12-06 09:02:56 +03:00
Azat Khuzhin
1167313961 stress: allow memory overcommit for one more query 2021-12-06 09:02:56 +03:00
Azat Khuzhin
7063ea3ee2 stress: add make_query_command() helper 2021-12-06 09:02:56 +03:00
Azat Khuzhin
adec690166 stress: SYSTEM DROP MARK CACHE 2021-12-06 09:02:56 +03:00
Azat Khuzhin
d3a7aed005 stress: tune memory_profiler_step too 2021-12-06 09:02:56 +03:00
Azat Khuzhin
169941c5d0 stress: allow 100MiB of memory to overcommit for SHOW/DROP DATABASE 
CI: https://s3.amazonaws.com/clickhouse-test-reports/32019/b6290ae00223fc91d514b82de25e195ab34f8bca/stress_test__thread__actions_.html
 2021-12-06 09:02:56 +03:00
Azat Khuzhin
445b0983ea stress: ignore server memory limits for hung check 
In [1] hung check failed because the server was under memory pressure.

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/32019/b6290ae00223fc91d514b82de25e195ab34f8bca/stress_test__undefined__actions_.html
 2021-12-06 09:02:56 +03:00
Azat Khuzhin
3e96b28843 perf: convert killall to pkill (since killall has some magic, see -e option) 2021-12-03 10:56:42 +03:00
Azat Khuzhin
fb0cc62509 perf: fix waiting of the server after running tests 
killall requires strict match, i.e. "clickhouse-server" not
"clickhouse":

    2021-12-03 05:24:56      + env kill -- -21700
    2021-12-03 05:24:56      kill: (-21700): No such process
    2021-12-03 05:24:56      + killall clickhouse
    2021-12-03 05:24:56      clickhouse: no process found
    2021-12-03 05:24:56      + echo Servers stopped.
    2021-12-03 05:24:56      Servers stopped.
    2021-12-03 05:24:56      + analyze_queries

    $ tail -n1 *-server-log.log
    ==> left-server-log.log <==
    2021.12.03 05:26:59.530647 [ 450 ] {} <Trace> SystemLog (system.asynchronous_metric_log): Flushed system log up to offset 1668052

    ==> right-server-log.log <==
    2021.12.03 05:27:20.873136 [ 466 ] {} <Trace> SystemLog (system.metric_log): Flushed system log up to offset 9605

    ==> setup-server-log.log <==
    2021.12.03 02:47:14.844395 [ 96 ] {} <Information> Application: Child process exited normally with code 0.

As you can see killall instantly fails with no such process, while this
cannot be true since it was there, and also according to logs there were
messages after running analyze_queries() from compare.sh

This should fix problems like in [1].

  [1]: https://clickhouse-test-reports.s3.yandex.net/32080/344298f4037f88b114b8e798bb30036b24be8f16/performance_comparison/report.html#fail1
 2021-12-03 10:55:35 +03:00
mergify[bot]
9c016d4e25
Merge branch 'master' into stress-test 2021-12-02 12:02:44 +00:00
taiyang-li
2f4e7e1d4e merge master 2021-12-02 19:48:21 +08:00
Azat Khuzhin
e976cd8691 Provide clickhouse binary w/o debug symbols (stripped) in fasttest 
- regular binary: 1.8G
- stripped: 343M (5x less)
- gzipped stripped: 92M (20x less)

This can be useful to check something fast.
 2021-12-01 21:50:03 +03:00
mergify[bot]
b4f43c59a2
Merge branch 'master' into stress-test 2021-12-01 10:12:22 +00:00
alesapin
b4d4295c16 Followup 2021-11-30 13:24:04 +03:00
alesapin
085a30b98e Fixup 2021-11-30 13:22:50 +03:00
alesapin
bccd927aec Make stress tests slightly more stable 2021-11-30 13:19:03 +03:00
Raúl Marín
6a04dcb492 Remove libc-headers 2021-11-29 19:55:57 +01:00
Mikhail f. Shiryaev
e97e172dd8
Merge pull request #31822 from ClickHouse/rollback-28016 
Rollback changes from #28016: ru.archive.ubuntu.com
 2021-11-29 10:27:59 +01:00
mergify[bot]
e6f5792ba3
Merge branch 'master' into stress-test 2021-11-29 09:01:33 +00:00
Alexey Milovidov
0f465a971e Remove some old options 2021-11-27 03:36:29 +03:00
mergify[bot]
ae0b5df601
Merge branch 'master' into stress-test 2021-11-26 12:48:03 +00:00
Mikhail f. Shiryaev
5408d6232f
Make archive.ubuntu.com mirror configurable for build 2021-11-26 10:55:41 +01:00
alesapin
3562264404
Merge pull request #31803 from ClickHouse/faster_tests 
Shorter stress and fuzzer tests
 2021-11-26 10:17:26 +03:00
taiyang-li
d35e2a1c83
Merge branch 'master' into hive_table 2021-11-26 11:44:50 +08:00
alexey-milovidov
6b8f7ad03f
Merge pull request #31798 from ClickHouse/fix-31748 
Fix stylecheck for tests/ci/push_to_artifactory.py
 2021-11-26 00:11:32 +03:00
alexey-milovidov
6017b982d9
Merge pull request #31780 from Algunenano/fix_performance_report_queries 
Improve how queries are output in the performance dashboard
 2021-11-25 22:57:29 +03:00
alesapin
71ae890c59
Merge pull request #31544 from azat/ci-actions-fixes 
Remove duplicated not compressed logs from CI (actions)
 2021-11-25 18:43:44 +03:00
alesapin
5815429059 Shorter stress and fuzzer tests 2021-11-25 17:54:50 +03:00
Mikhail f. Shiryaev
16703c2824
Fix stylecheck for tests/ci/push_to_artifactory.py 2021-11-25 14:41:08 +01:00
alexey-milovidov
e8b29cab6e
Update report.py 2021-11-25 15:37:00 +03:00
mergify[bot]
ce0c112586
Merge branch 'master' into stress-test 2021-11-25 11:09:26 +00:00
Raúl Marín
a6c34fdbc9 Improve how queries are output in the performance dashboard 2021-11-25 11:22:47 +01:00
taiyang-li
72f60cceb9
Merge branch 'master' into hive_table 2021-11-25 17:33:26 +08:00
alesapin
c86b43c7ef
Merge pull request #31685 from ClickHouse/tavplubix-patch-5 
Try fix OOMs with TSAN
 2021-11-25 11:34:05 +03:00
alesapin
58fad90351
Merge branch 'master' into add_performance_tests 2021-11-25 10:53:10 +03:00
alesapin
cf2f019caa Disable full run in perf tests 2021-11-24 23:05:27 +03:00
Alexey Milovidov
21f112b30e Merge branch 'better-crosscompiling-freebsd' into hermetic-builds 2021-11-24 22:43:30 +03:00
Alexey Milovidov
92031e6b31 Fix PVS-Studio 2021-11-24 20:25:11 +03:00
tavplubix
82e4f941a4
Update Dockerfile 2021-11-24 16:00:15 +03:00
tavplubix
556ccd0bc9
Update Dockerfile 2021-11-24 12:47:44 +03:00
taiyang-li
89dcef69d5 merge master 2021-11-24 14:38:04 +08:00
Alexey Milovidov
51ee8fdbb0 More easy cross-compiling for FreeBSD 2021-11-24 00:32:04 +03:00
alexey-milovidov
61454ffbcb
Merge branch 'master' into hermetic-builds 2021-11-24 00:25:16 +03:00
alesapin
42c77323e8
Merge branch 'master' into add_performance_tests 2021-11-23 22:14:32 +03:00
Kruglov Pavel
18e60919c5
Update run.sh 2021-11-23 16:49:59 +03:00
alexey-milovidov
341f705419
Merge pull request #30599 from ClickHouse/zlib-ng-own-cmake 
Use our own CMakeLists for `zlib-ng`, `cassandra`, `mariadb-connector-c` and `xz`, `re2`, `sentry`, `gsasl`, `arrow`, `protobuf`.
 2021-11-23 13:33:29 +03:00
alesapin
2396f73e08 Fix performance tests comparison 2021-11-23 11:34:10 +03:00
alesapin
62b2eb6694 Merge branch 'master' into add_performance_tests 2021-11-23 11:26:45 +03:00
lgbo
996d7125c0
Merge branch 'master' into hive_table 2021-11-23 10:19:02 +08:00
Constantine Peresypkin
fd551ad94c fix kerberized_hadoop image 
current image uses non-existing centos6 repos
added direct fetch of actual rpm packages

fixed a typo in docker/images.json: duplicate entry
 2021-11-22 17:32:19 +02:00
taiyang-li
d22854e82b add compile swiych USE_HIVE 2021-11-22 10:52:10 +08:00
Alexey Milovidov
d604cf5573 Remove the infamous "unbundled" build 2021-11-21 13:58:26 +03:00
Azat Khuzhin
d0ecdb617e Remove duplicated not compressed logs from CI (actions) 
Right now we cannot remove that compression since non-github actions
check will upload non compressed files then, but once old CI will be
removed it seems to be possible to remove that manual compression.
 2021-11-19 21:33:48 +03:00
alesapin
9e440988cf Add datasets 2021-11-19 16:10:49 +03:00
taiyang-li
ab64a17104 fix fasttest 2021-11-19 16:20:14 +08:00
taiyang-li
e8644807fe merge master and solve conflict 2021-11-19 15:01:58 +08:00
alesapin
130c09f348 Fix search for old version 2021-11-18 13:38:59 +03:00
lgbo-ustc
9902ccefc5 Add integration tests for hive query 2021-11-18 16:17:49 +08:00
alesapin
d0ec0c5dc8 Merge branch 'master' into add_performance_tests 2021-11-18 10:17:20 +03:00
alesapin
b74f9e7047 Merge branch 'master' into add_performance_tests 2021-11-18 10:12:46 +03:00
Nikolay Degterinsky
fc9ef14a73
Merge pull request #29219 from evillique/log_lz4_streaming 
Add LZ4 stream compression of logs
 2021-11-17 21:22:57 +03:00
Nikolay Degterinsky
eafd3ddfac Add lz4 stream compression 2021-11-16 22:30:29 +00:00
tavplubix
b1079ae309
Update run.sh 2021-11-16 20:03:50 +03:00
mergify[bot]
923874060a
Merge branch 'master' into stress-test 2021-11-16 16:42:50 +00:00
tavplubix
224a4c1b6b
Update run.sh 2021-11-16 17:45:37 +03:00
alesapin
7f09941c72 Add performance comparison check 2021-11-16 14:16:10 +03:00
Kruglov Pavel
6180aedab6
Update download_previous_release 2021-11-15 23:16:35 +03:00
Kruglov Pavel
87df3c8272
Fix 2021-11-15 15:03:48 +03:00
alexey-milovidov
f4fda976ef
Merge pull request #31063 from azat/perf-spikes 
perf: switch *_log tables to Memory engine (attempt to reduce cache misses)
 2021-11-14 05:13:12 +03:00
Azat Khuzhin
48b1b7b7ec Fix test_kafka_insert_avro by pinning avro version 
New avro version has a code that does not allow to use BytesIO [1].

  [1]: https://github.com/apache/avro/commit/e8c61ba4f0#diff-ca721cc66db80ff81dcd7b8ad1bb0e83b2e5c2de80c207ebec79f5495d38db98R318-R319
 2021-11-13 12:01:13 +03:00
Azat Khuzhin
40be97c1dd Improve fuzzer report in case of fuzzer killed 
This will avoid Exception from fuzzer in description like in [1], that
someone may think that is related to the failure, which is obviously
don't in case it was KILLed.

  [1]: https://clickhouse-test-reports.s3.yandex.net/31259/a49e10c78860a451ff9cc39df6049c88234d507a/fuzzer_ubsan/report.html#fail1
 2021-11-12 01:01:10 +03:00
mergify[bot]
045bd40070
Merge branch 'master' into perf-spikes 2021-11-11 07:07:28 +00:00
taiyang-li
af78116c3f
Merge branch 'master' into hive_table 2021-11-11 11:52:28 +08:00
alesapin
0b28474f09
Merge pull request #31244 from nikitamikhaylov/disable-fuzz-builds 
Disable fuzzer builds in CI
 2021-11-10 16:42:08 +03:00
Nikita Mikhaylov
99f78472bc Disabled 2021-11-10 11:38:26 +00:00
Kseniia Sumarokova
5f9b35b982
Update docker_compose_postgres.yml 2021-11-10 10:36:59 +03:00
taiyang-li
26988ab757
Merge branch 'master' into hive_table 2021-11-10 11:01:03 +08:00
Kruglov Pavel
08af98b4a1
Update download_previous_release 2021-11-09 16:34:13 +03:00
Kseniia Sumarokova
5eef7e0e45
Update docker_compose_postgres.yml 2021-11-09 15:13:01 +03:00
taiyang-li
5dbbe4b9c5 fix build error 2021-11-09 13:14:53 +08:00
Kruglov Pavel
03f1e3c67d
Update run.sh 2021-11-08 14:02:36 +03:00
Azat Khuzhin
9e622b5add perf: switch *_log tables to Memory engine (attempt to reduce cache misses) 
trace_log/query_log from performance tests shows (for
cases when prewarm query fails with timeout, 15sec) excessive
writeTraceInfo() in trace_log and QueryProfilerRuns in query_log, but
this is not the root cause of the timeout, but consequence.

Also query_log shows that on failures the following profile events has
significantly higher values:
- PerfLocalMemoryMisses (6.3x more)
- PerfLocalMemoryReferences (7x more)
- PerfDataTLBMisses (6.9x more)
- PerfInstructionTLBMisses (6.4x more)

During looking at performance tests logs I noticed that once the prewarm
query fails other server (left/right) was merging (MergeTree) something
in *_log tables.

But, using MergeTree for *_log in performance tests is useless, since
anyway environment for performance tests uses ramdrive.

And so MergeTree merges just increase overhead.

Eventually I expect that this should decrease extra memory referencing
and so this should decrease cache/TLB misses.

CI: https://clickhouse-test-reports.s3.yandex.net/30886/c504e0c08df7a926bb479a1d297f326f5c48a32f/performance_comparison/report.html#fail1

v2: <partition_by remove="remove"/>
 2021-11-04 09:26:46 +03:00
mergify[bot]
8ef9e61e36
Merge branch 'master' into stress-test 2021-11-03 08:28:43 +00:00
Azat Khuzhin
ff19d22305 perf: remove over *_log tables that does not exported into artifacts 2021-11-03 08:29:18 +03:00
Azat Khuzhin
837fe8554f perf: remove zookeeper to avoid configuring DDLWorker 2021-11-03 08:29:18 +03:00
Azat Khuzhin
acef85b17d perf: cleanup server overrides 2021-11-03 08:29:18 +03:00
Azat Khuzhin
afd960db4e perf: remove redundant query profiler set queries 2021-11-03 08:09:43 +03:00
robot-clickhouse
4289336c27 Auto version update to [21.12.1.1] [54457] 2021-11-02 00:56:45 +03:00
Kruglov Pavel
9a1275cb10
Merge pull request #30178 from Avogar/tsv-csv 
Refactor and improve TSV, CSV, JSONCompactEachRow, RowBinary formats. Fix bugs in formats
 2021-11-02 00:38:30 +03:00
alesapin
dbfa17aabf Add correct S3 2021-11-01 13:32:56 +03:00
avogar
4e63311bb6 Try fix perf test 2021-11-01 12:13:29 +03:00
alesapin
54e404de32 Add github to known hosts in docs release 2021-11-01 10:33:52 +03:00
alesapin
ed3d707513
Merge pull request #30839 from ClickHouse/qoega-docker-docs 
Docs check on github actions
 2021-10-31 20:29:04 +03:00
alexey-milovidov
15b9d65221
Merge pull request #30662 from azat/conf-clickhouse 
Switch everything left from `<yandex>` to `<clickhouse>`
 2021-10-31 14:56:51 +03:00
alesapin
d872eae809 Better 2021-10-29 21:43:03 +03:00
alesapin
4c645f3ef6 Add docs release 2021-10-29 21:43:03 +03:00
alesapin
1a13a741f9 Fixes in docs 2021-10-29 21:43:03 +03:00
alesapin
6c8ba21c52 Add docs check image 2021-10-29 21:43:03 +03:00
alesapin
11f71fffe1 Tabs to spaces 2021-10-29 21:43:03 +03:00
alesapin
afcadc95a7 Add docs check to documentation 2021-10-29 21:43:03 +03:00
Yatsishin Ilya
faeb4ef869 Add docker container to build docs 2021-10-29 21:43:03 +03:00
tavplubix
059c1ebf36
Merge pull request #30847 from ClickHouse/fix_30802 
Disable check for ZooKeeper session uptime by default
 2021-10-29 20:18:25 +03:00
Alexander Tokmakov
d095cfe4c1 disable check for zk session uptime by default 2021-10-29 14:45:09 +03:00
alesapin
4eca493ab8 Fix fuzzer on master 2021-10-29 13:47:29 +03:00
alesapin
11b70a285c
Merge pull request #30795 from ClickHouse/make_fuzzer_check_less_hardcoded 
Trying to make fuzzer check less hardcoded
 2021-10-29 10:21:36 +03:00
alesapin
ddac7a51e2
Merge pull request #30729 from ClickHouse/trying_actions 
Add functional tests to github actions
 2021-10-29 09:49:31 +03:00
mergify[bot]
65df198e9b
Merge branch 'master' into conf-clickhouse 2021-10-28 22:19:42 +00:00
alesapin
ffc8dbd085
Update run-fuzzer.sh 2021-10-28 23:06:04 +03:00
alesapin
313584e553
Update run-fuzzer.sh 2021-10-28 19:12:47 +03:00
mergify[bot]
a94b8902dd
Merge branch 'master' into trying_actions 2021-10-28 15:57:28 +00:00
alesapin
0651a1f0e9
Update run-fuzzer.sh 2021-10-28 17:03:40 +03:00
alesapin
542196128b Trying to make fuzzer check less hardcoded 2021-10-28 12:29:50 +03:00
alesapin
c0b2f92123 Add stateful test 2021-10-28 12:26:59 +03:00
Azat Khuzhin
9eef2ad0c2 perf: reduce number of times the query will run 
Before this patch:

- upstream/master and PR's *with* perf tests or pef scripts changes:
  --runs=13 --max-queries=0

- PRs *without* perf changes:
  --runs=7 --max-queries=20

- PRs w/ only perf tests changes:
  --runs-13 --max-queries=0 <list of perf tests>

After:

- upstream/master and PR's *with* perf tests changes:
  --runs=13 --max-queries=0

- PRs *without* perf changes:
  --runs=7 --max-queries=10

- PRs w/ only perf tests changes:
  --runs-13 --max-queries=0 <list of perf tests>

So to underline, now we will not look at perf scripts changes anymore,
and we will also decrease number of random queries to run to 10.
 2021-10-27 23:20:51 +03:00
Azat Khuzhin
7957079f95 perf: disable query/memory profiling for non prewarm queries 2021-10-27 23:20:51 +03:00
Azat Khuzhin
190459db33 perf: add queries with errors during process too to the perf artifacts 
Sometimes queries fails with timeout, and you will not get any perf
metrics for it before this patch.

Example: https://clickhouse-test-reports.s3.yandex.net/30611/5ff6c5536558821824d5fdf25a75729e5b82060d/performance_comparison/report.html#partial-queries.flat_dictionary.5
 2021-10-27 23:20:51 +03:00
Azat Khuzhin
722d528218 perf: pin clickhouse-driver to 0.2.1 for tzlocal <0.3 to avoid warning 
CI report [1]:

    /usr/local/lib/python3.6/dist-packages/clickhouse_driver/columns/datetimecolumn.py:199: PytzUsageWarning: The zone attribute is specific to pytz's interface; please migrate to a new time zone provider. For more details on how to do so, see https://pytz-deprecation-shim.readthedocs.io/en/latest/migration.html

  [1]: https://clickhouse-test-reports.s3.yandex.net/30626/dfc85841134aa96b4e04c401462898eb305e8657/performance_comparison/report.html#changes-in-performance.format_date_time.1
 2021-10-27 23:20:51 +03:00
mergify[bot]
d9d9d38e4f
Merge branch 'master' into stress-test 2021-10-27 19:00:31 +00:00
Azat Khuzhin
90ea2de9bb Add unidiff into style image 2021-10-27 10:03:24 +03:00
Azat Khuzhin
b41c77f4f8 Add boto3 to style image 2021-10-27 10:03:24 +03:00
Azat Khuzhin
944437caf6 Add PyGithub into style image 2021-10-27 10:03:23 +03:00
Nikita Mikhaylov
a4100cc980 Build local in CI in fuzz mode 2021-10-26 17:45:41 +00:00
Nikita Mikhaylov
aae30a9e37
Fix fuzzer build (#30344) 2021-10-26 20:30:34 +03:00
Azat Khuzhin
da90df5658 docker: switch <yandex> to <clickhouse> 2021-10-26 08:53:14 +03:00
Azat Khuzhin
8f015764a7 Add metadata for *_log into tests artifacts 
This will allow to use artifacts as-is, i.e.:

    clickhouse-local --path /path/to/log/artifact

v2: convert database to Ordinary to avoid extra symlinks dependencies.
 2021-10-23 20:17:01 +03:00
mergify[bot]
943a1cbba0
Merge branch 'master' into stress-test 2021-10-21 04:41:23 +00:00
alesapin
9e8dfdb089
Merge pull request #28851 from ClickHouse/trying_actions 
Trying github actions
 2021-10-20 12:55:53 +03:00
alesapin
6d8853698b Merge branch 'master' into trying_actions 2021-10-19 17:45:37 +03:00
vdimir
acf416900d
Minor fix in clickhouse/kerberized-hadoop Dockerfile 2021-10-19 16:45:03 +03:00
vdimir
1d78f1c63b
Fix ca-bundle.crt in clickhouse/kerberized-hadoop 2021-10-19 16:40:06 +03:00
vdimir
36bb4033ba
Whitespace change in kerberized_hadoop/Dockerfile 2021-10-19 12:44:55 +03:00
Alexey Milovidov
2cef9983fa Remove recursive submodules 2021-10-18 02:29:04 +03:00
Alexey Milovidov
f02b61e30f Merge branch 'master' into hermetic-builds 2021-10-17 07:02:37 +03:00
mergify[bot]
f29ea07b1c
Merge branch 'master' into clickhouse-test-http-interface 2021-10-13 16:46:26 +00:00
tavplubix
a16ce6b54b
Merge pull request #30030 from ClickHouse/tavplubix-patch-3 
Attach trace_log to stateless tests report as is
 2021-10-13 13:21:39 +03:00
Alexey Milovidov
b420bc19fb Merge branch 'master' into hermetic-builds 2021-10-13 01:08:23 +03:00
Dmitry Novik
fa9cdd5c5f Use original whitespaces in test_results.tsv 2021-10-12 23:31:31 +03:00
Dmitry Novik
0507391010 Do not stop CI tests processing if clickhouse-test return 1 2021-10-12 23:30:07 +03:00
Dmitry Novik
a30573fc0a Add FAIL message to test_results.tsv 2021-10-12 23:29:15 +03:00
Azat Khuzhin
071a5af96e Revert "docker: add pandas/clickhouse_driver into test images" 
This reverts commit e07a6f3fc0.
 2021-10-12 21:06:26 +03:00
alexey-milovidov
9a1f930b2f
Merge pull request #30010 from ClickHouse/ppc64le 
Add cross-build for PowerPC 64le
 2021-10-12 16:17:41 +03:00
tavplubix
f3c9f4be6c
Update run.sh 2021-10-12 14:38:41 +03:00
alesapin
2b3a0c892d
Merge pull request #29992 from ClickHouse/more_timeouts_in_scripts 
More timeouts in test scripts
 2021-10-12 10:14:21 +03:00
Alexey Milovidov
bdd06279a4 Fix PVS Studio 2021-10-12 05:43:53 +03:00
Alexey Milovidov
d0f7a10c6a Fix fasttest 2021-10-12 03:34:07 +03:00
Alexey Milovidov
75547e64a5 Add to packager 2021-10-12 02:10:58 +03:00
alesapin
3ae960e04b Review fixes 2021-10-11 16:40:12 +03:00
alesapin
9ad919d91a More timeouts in stress test 2021-10-11 13:25:54 +03:00
Alexey Milovidov
0dbe7c4036 Remove old toolchain 2021-10-11 07:02:19 +03:00
alexey-milovidov
268c155b7d
Merge pull request #29856 from azat/clickhouse-test-python-client 
Rewrite clickhouse-test to use python clickhouse_driver
 2021-10-09 17:58:40 +03:00
alexey-milovidov
f6a0983068
Merge pull request #29781 from azat/fuzzer-server-pid-v2 
Simplify obtaining of server pid in fuzzer tests
 2021-10-08 00:35:25 +03:00
Azat Khuzhin
e07a6f3fc0 docker: add pandas/clickhouse_driver into test images 2021-10-08 00:09:37 +03:00
Kruglov Pavel
fd7d1131ec
Update run.sh 2021-10-06 13:17:48 +03:00
mergify[bot]
6eec0d76a7
Merge branch 'master' into update-minio 2021-10-06 05:45:18 +00:00
Azat Khuzhin
aeb670dbe0 Simplify obtaining of server pid in fuzzer tests 2021-10-05 21:05:49 +03:00
Nikita Mikhaylov
a880cce0de better 2021-10-05 13:26:29 +00:00
mergify[bot]
0910a8ec74
Merge branch 'master' into stress-test 2021-10-05 09:46:18 +00:00
Kseniia Sumarokova
ead99b5911
Update docker_compose_minio.yml 2021-10-05 11:39:08 +03:00
kssenii
4962677a8b Update minio 2021-10-05 08:18:16 +00:00
Nikita Mikhaylov
731d6c5755
Merge pull request #29513 from azat/fuzzer-fix-server-pid 
Fix server pid (hence exit code and attaching with gdb) in fuzzer tests
 2021-10-04 19:59:24 +03:00
Kruglov Pavel
31302afe81 Update run.sh 2021-10-04 16:35:50 +03:00
avogar
821daa21da Update run.sh 2021-10-04 16:35:50 +03:00
Pavel Kruglov
6fc043026e Update run.sh 2021-10-04 16:35:50 +03:00
Pavel Kruglov
2db11bc6c5 Add backward compatibility check in stress test 2021-10-04 16:34:14 +03:00
Azat Khuzhin
95d8133aaa Cleanup unbundled image 
No need to install specific boost packages, when all boost is installed.
 2021-10-04 00:58:29 +03:00
Azat Khuzhin
74a7a0603d test/fuzzer: print stacktrace 
To get the stacktrace for errors w/o it [1]:

    SET max_parser_depth = 4000

    Got boring AST
    Code: 306. DB::Exception: Maximum parse depth (1000) exceeded. Consider rising max_parser_depth parameter. (TOO_DEEP_RECURSION)

  [1]: https://clickhouse-test-reports.s3.yandex.net/29513/769bfbe71f22c392458eced655080a0bbd442e02/fuzzer_ubsan/fuzzer.log
 2021-10-02 12:49:44 +03:00
Azat Khuzhin
769bfbe71f Fix server pid in fuzzer tests 
Previously it was the pid of the subshell 40 while it should be the pid
of the clickhouse-server 39:

Here we see that the server pid is 39:

    2021-09-28 11:02:34	 + pgrep -f clickhouse-server
    2021-09-28 11:02:34	 39

Here we see that the 40 is the pid of subshell:

    2021-09-28 11:02:45	 ch/docker/test/fuzzer/run-fuzzer.sh: line 90:    39 Killed                  clickhouse-server --config-file db/config.xml -- --path db 2>&1
    2021-09-28 11:02:45	         40 Done                    | tail -100000 > server.log

And here we see that server_pid variable is 40:

    2021-09-28 11:02:45	 + server_exit_code=0
    2021-09-28 11:02:45	 + wait 40

v2: wait in background to call wait in foreground and ensure that the
process is alive, since w/o job control this is the only way to obtain
the exit code
 2021-10-01 21:05:30 +03:00
alesapin
53fce1411c
Merge pull request #29597 from ClickHouse/fix-docker-image 
Fix docker/test/util declaration
 2021-10-01 13:43:17 +03:00
Alexey Milovidov
4271f8c738 Merge remote-tracking branch 'origin/master' into yandex-to-clickhouse-in-configs 2021-10-01 01:33:58 +03:00
Dmitry Novik
ad3c3f72e4 Fix docker/test/util declaration 2021-10-01 01:04:06 +03:00
Alexey Milovidov
2ec75b824b Revert "repo.clickhouse.tech is not 100% ready" 
This reverts commit 8616ba62e4.
 2021-09-29 02:43:20 +03:00
Dmitry Novik
29587c5013 Pass all parameters to process_functional_tests_result.py 2021-09-28 20:38:41 +03:00
Dmitry Novik
6b12869f6e Fix path to executable 2021-09-28 15:11:18 +03:00
Dmitry Novik
4ce12a7685 Fixup 2021-09-28 11:50:28 +03:00
alesapin
8e25144bbb Merge branch 'master' into trying_actions 2021-09-28 11:08:13 +03:00
Dmitry Novik
7755b3688f Fix docker images build 2021-09-28 01:07:53 +03:00
Dmitry Novik
5418ea83fa Fix clickhouse/test-util Dockerfile 2021-09-27 17:38:25 +03:00
Dmitry Novik
d9ce2c2d99 Update images.json 2021-09-27 15:59:20 +03:00
Dmitry Novik
4e6c945fdf Rename fasttest output file 2021-09-27 13:09:57 +03:00
mergify[bot]
dbb924171d
Merge branch 'master' into fasttest-tsv 2021-09-27 07:48:49 +00:00
mergify[bot]
64669a28ce
Merge branch 'master' into yandex-to-clickhouse-in-configs 2021-09-26 22:59:23 +00:00
Alexey Milovidov
dc7eb5c1ae Fix ugly typo 2021-09-26 03:29:32 +03:00
Alexey Milovidov
fb7ce28198 Merge branch 'master' into yandex-to-clickhouse-in-configs 2021-09-26 02:10:11 +03:00
Alexey Milovidov
90e74aff70 Fix performance test 2021-09-26 02:08:46 +03:00
Dmitry Novik
98df97f5d9 Fix docker image 2021-09-24 13:48:28 +03:00
Dmitry Novik
77dfa44256 Generate test_results.tsv for fasttest 2021-09-24 12:16:08 +03:00
Alexey Milovidov
8616ba62e4 repo.clickhouse.tech is not 100% ready 2021-09-24 04:41:59 +03:00
mergify[bot]
88b7680eeb
Merge branch 'master' into trying_actions 2021-09-23 15:48:57 +00:00
Alexey Milovidov
e9e77b4403 .tech -> .com 2021-09-22 03:22:57 +03:00
alexey-milovidov
0880ae2883
Merge branch 'master' into yandex-to-clickhouse-in-configs 2021-09-21 20:17:28 +03:00
tavplubix
8adfb9b593
Stop ThreadFuzzer before hung check (#29167) 
* stop ThreadFuzzer before hung check

* fix

* fix
 2021-09-20 17:23:10 +03:00
Alexey Milovidov
147ef79670 Change website URL, part 1 2021-09-20 01:56:28 +03:00
Alexey Milovidov
bdc99b6c91 Change Alpine Dockerfile 2021-09-20 01:53:52 +03:00
Alexey Milovidov
ab44e0addb Change Docker org name 2021-09-20 01:52:21 +03:00
Alexey Milovidov
510540c7d7 Change Doker Hub URL 2021-09-20 01:45:18 +03:00
Alexey Milovidov
e513a5db32 Change <yandex> to <clickhouse> in configs 2021-09-20 01:38:53 +03:00
alesapin
00d9a62d6c Missclick 2021-09-17 14:17:50 +03:00
alesapin
5d35ba7d93 One more time 2021-09-17 13:41:29 +03:00
alesapin
68c7f8638a update docker image 2021-09-17 13:34:45 +03:00
alesapin
71dcf03a0f Create changed image 2021-09-15 20:35:15 +03:00
Nikita Mikhaylov
4b3c0a95d1
Merge pull request #28046 from syominsergey/fix-build-clang-13-2 
[WIP] Improve support for build with clang-13
 2021-09-13 12:11:38 +03:00
Vitaly Baranov
512786346c
Merge pull request #28909 from vitlibar/convert-skip-list-into-first-line-comments 
Convert skip_list .json into first line comments
 2021-09-13 03:37:39 +03:00
Sergei Semin
d3b3e8607c fix tabs 2021-09-12 19:51:47 +03:00
Sergei Semin
12e24a23d3
Merge branch 'master' into fix-build-clang-13-2 2021-09-12 19:30:55 +03:00
Vitaly Baranov
daa5979347 Remove unnecessary files. 2021-09-12 17:15:28 +03:00
Vitaly Baranov
bbb192ee85 Convert skip_list.json into first line comments. 2021-09-12 17:15:23 +03:00
alexey-milovidov
60f76d9254
Merge pull request #28759 from nikitamikhaylov/enable-gcc-11 
Switch CI to GCC-11
 2021-09-12 15:56:57 +03:00
alexey-milovidov
ea13a8b562
Merge pull request #28659 from myrrc/improvement/tostring_to_magic_enum 
Improving CH type system with concepts
 2021-09-12 15:26:29 +03:00
Sergei Semin
b7358636c1 remove duplicated libraries 2021-09-11 18:10:18 +03:00
Mike Kot
c4385a2c40 Simplifying exception messages 2021-09-11 00:40:18 +02:00
mergify[bot]
e3b3439535
Merge branch 'master' into fix-build-clang-13-2 2021-09-10 12:05:54 +00:00
alesapin
6243b64ff5
Merge pull request #28656 from ClickHouse/move_docker_images_to_separate_repo 
Move docker images to a separate repo
 2021-09-10 10:24:21 +03:00
mergify[bot]
937eeb9fed
Merge branch 'master' into enable-gcc-11 2021-09-09 22:44:10 +00:00
Nikita Mikhaylov
e575cc88fc Better 2021-09-09 21:11:07 +00:00
Sergei Semin
ffc529feb1 use clang-13 in pvs docker image 2021-09-09 23:51:47 +03:00
Sergei Semin
b739d1cc92 make changes analog to https://github.com/ClickHouse/ClickHouse/pull/20495/files but for clang-13 2021-09-09 23:51:47 +03:00
Sergei Semin
63b3b90d1e up LLVM_VERSION and don't use script for llvm setup 2021-09-09 23:51:47 +03:00
Sergei Semin
ff635bcab8 add build with clang-13 2021-09-09 23:51:47 +03:00
alesapin
2359a222ff Fix more tests 2021-09-09 23:29:42 +03:00
robot-clickhouse
fc4f119417 Auto version update to [21.11.1.1] [54456] 2021-09-09 12:03:29 +03:00
alesapin
1297e95e4a Revert "Revert jdbc-bridge image" 
This reverts commit 63cc735b8f.
 2021-09-09 10:45:04 +03:00
alesapin
63cc735b8f Revert jdbc-bridge image 2021-09-09 10:42:59 +03:00
alesapin
4d49e6da58 Fix build image name 2021-09-09 10:42:59 +03:00
Nikita Mikhaylov
107710e618 save 2021-09-08 15:06:15 +00:00
mergify[bot]
64fb384656
Merge branch 'master' into global-merge-executor 2021-09-08 13:15:38 +00:00
alesapin
cbbc5b2ee5 Fix some broken images 2021-09-08 12:58:17 +03:00
alesapin
bdd26f2a19 Merge branch 'master' into move_docker_images_to_separate_repo 2021-09-08 12:36:36 +03:00
alesapin
3b9dae8718
Merge pull request #28616 from ClickHouse/qoega-try-llvm-12 
Merging #20495
 2021-09-08 11:43:46 +03:00
Vitaly Baranov
70c6623036
Merge branch 'master' into governance/session_log 2021-09-07 10:12:54 +03:00
Nikita Mikhaylov
29a18d8d10 Try to fix unbundled 2021-09-06 15:58:08 +00:00
alesapin
861258ecc0 Simplier 2021-09-06 15:06:41 +03:00
alesapin
714b2a3c19 More 2021-09-06 14:33:49 +03:00
alesapin
0eac4529e0 Preparation 2021-09-06 14:33:39 +03:00
alesapin
40d4d64a65 Fix PVS Image 2021-09-05 15:50:25 +03:00
alesapin
4df6fa1ae7 Remove strange changes 2021-09-04 16:09:12 +03:00
alesapin
a879c907a2 Update PVS hashsum 2021-09-04 16:00:42 +03:00
mergify[bot]
6ccf905133
Merge branch 'master' into try-llvm-12 2021-09-03 16:13:13 +00:00
alesapin
4fe3909d74
Update Dockerfile 2021-09-03 14:01:47 +03:00
alesapin
69e7e1dff7
Update Dockerfile 2021-09-03 14:00:57 +03:00
Nikolay Degterinsky
53b6d918eb
Merge pull request #25310 from bharatnc/ncb/import-export-lz4 
LZ4 compression for import/export
 2021-09-01 21:12:50 +03:00
alesapin
7e2cb7c8ac
Merge pull request #27538 from ClickHouse/stress-thread-fuzzer 
Enable Thread Fuzzer in Stress test
 2021-09-01 10:26:19 +03:00
Nikita Mikhaylov
a738b145ba fix shellcheck 2021-08-31 12:54:45 +00:00
Nikita Mikhaylov
a5d4060df8 Better waiting for minio to start 2021-08-31 12:54:45 +00:00
vdimir
56a3675591 Adjust docker/test/stateless/setup_minio.sh for local run 2021-08-31 12:54:45 +00:00
Nikita Mikhaylov
c2c9bea19e Update setup_minio.sh 2021-08-31 12:54:45 +00:00
Nikita Mikhaylov
5f7a1bab94 better 2021-08-31 12:54:45 +00:00
Nikita Mikhaylov
cc73ca17d3 Debug 2021-08-31 12:54:45 +00:00
Nikita Mikhaylov
dbe13ffc0f Fix tests 2021-08-31 12:54:45 +00:00
Nikita Mikhaylov
083ca0f283 Better 2021-08-31 12:54:45 +00:00
Nikita Mikhaylov
4cdc978636 Move dowloading to dockerfile 2021-08-31 12:54:45 +00:00
Nikita Mikhaylov
92446b1b88 Better 2021-08-31 12:54:45 +00:00
Nikita Mikhaylov
4a88e5f281 Better 2021-08-31 12:54:45 +00:00
Nikita Mikhaylov
1adbb2db18 Update setup_minio.sh 2021-08-31 12:54:45 +00:00
Nikita Mikhaylov
daece09eab chmod +x 2021-08-31 12:54:45 +00:00
Nikita Mikhaylov
c4bf47d4a3 Better 2021-08-31 12:54:45 +00:00
Nikita Mikhaylov
26efa7184d Added minio 2021-08-31 12:54:45 +00:00
Nikita Mikhaylov
e41d1cff1d Fix test 2021-08-31 12:54:45 +00:00
Nikita Mikhaylov
eed958495f Better 2021-08-31 12:54:45 +00:00
Nikita Mikhaylov
fcfda4b983 Fix tests and update ya.make 2021-08-31 12:54:45 +00:00
Nikita Mikhaylov
d7a627e5a9 Added test to skip list 2021-08-31 12:54:06 +00:00
mergify[bot]
b81d479de4
Merge branch 'master' into stress-thread-fuzzer 2021-08-31 11:56:30 +00:00
Nikolay Degterinsky
e8713a7d7e Merge branch 'master' into ncb/import-export-lz4 2021-08-31 13:32:39 +03:00
alesapin
525999145e
Merge pull request #26791 from ClickHouse/async-reads 
Experiment with asynchronous readers
 2021-08-31 13:17:45 +03:00
Vladimir C
3c2cee392a
Merge pull request #28350 from vdimir/fuzz-j2-sed-typo 2021-08-31 11:04:07 +03:00
Vladimir C
4cbc1aba19
Disable SC2001 shellcheck in docker/test/fuzzer/run-fuzzer.sh 2021-08-31 11:02:52 +03:00
Alexey Milovidov
143744f5a7 Fix perf test 2021-08-31 02:53:55 +03:00
Vasily Nemkov
c902afddde Added system.session_log table 
Which logs all the info about LogIn, LogOut and LogIn Failure events.
Additional info that is logged:
- User name
- event type (LogIn, LogOut, LoginFailure)
- Event date\time\time with microseconds
- authentication type (same as for IDENTIFIED BY of CREATE USER statement)
- array of active settings profiles upon login
- array of active roles upon login
- array of changed settings with corresponding values
- client address and port
- interface (TCP\HTTP\MySQL\PostgreSQL, etc.)
- client info (name, version info)
- optional LoginFailure reason text message.

Added some tests to verify that events are properly saved with all necessary info via following interfaces:
- TCP
- HTTP
- MySQL

Known limitations
- Not tested against named HTTP sessions, PostgreSQL and gRPC, hence those are not guaranteed to work 100% properly.
 2021-08-30 18:28:28 +03:00
Kseniia Sumarokova
00bea71610
Merge pull request #25251 from kssenii/disk-over-web-server 
ClickHouse over web server with static files
 2021-08-30 16:18:59 +03:00
vdimir
bc785bff25
Fix sed argument in test/fuzzer/run-fuzzer.sh 2021-08-30 14:25:08 +03:00
Azat Khuzhin
fb311199ce test/stress: fix patterns for filtering out Raft messages 2021-08-28 19:21:07 +03:00
Ilya Yatsishin
7f7ecb0c60
Merge branch 'master' into try-llvm-12 2021-08-26 18:58:49 +03:00
Nikita Mikhaylov
fc37817ada
Merge pull request #28150 from FArthur-cmd/fix_compression_26149 
Fix `Attempt to read after eof with enabled data compression`
 2021-08-26 17:17:09 +03:00
Ilya Yatsishin
0a9feade3d
Merge pull request #28079 from vdimir/fuzz-j2 
Use jinja template tests in fuzzer
 2021-08-26 10:53:45 +03:00
Nikita Mikhaylov
4a12a5fe91 Better 2021-08-25 21:29:28 +00:00
mergify[bot]
69d2a6080b
Merge branch 'master' into fix_compression_26149 2021-08-25 19:01:08 +00:00
Nikita Mikhaylov
fa95fa1517 Added a test 2021-08-25 17:42:57 +00:00
Nikita Mikhaylov
0615bf27d6
Merge pull request #27990 from nikitamikhaylov/fix-clang-tidy 
Build fuzzers in CI
 2021-08-25 18:33:41 +03:00
vdimir
92f073a435
Call generate-test-j2.py in run-fuzzer.sh before calculating NEW_TESTS_OPT 2021-08-25 16:05:16 +03:00
Nikolay Degterinsky
f722132f23 Merge branch 'master' into ncb/import-export-lz4 2021-08-25 14:59:17 +03:00
Nikita Mikhaylov
b8ef09a1f8 Fix build 2021-08-25 11:18:57 +00:00
Nikita Mikhaylov
ab22fdaa80 Fix build 2021-08-25 11:17:45 +00:00
Nikita Mikhaylov
6cece7d1d9
Update build.sh 2021-08-25 11:22:16 +03:00
vdimir
b9b0b17ad0
chmod +x docker/test/fuzzer/generate-test-j2.py 2021-08-25 11:20:42 +03:00
vdimir
0e37203dcd
Use jinja template tests in fuzzer 2021-08-25 11:18:30 +03:00
mergify[bot]
39f8ee024a
Merge branch 'master' into stress-thread-fuzzer 2021-08-25 01:27:19 +00:00
alexey-milovidov
47cc6c5afe
Merge pull request #28063 from vitlibar/set-fixed-version-of-tzlocal 
Set version of tzlocal to 2.1
 2021-08-25 02:48:08 +03:00
Nikita Mikhaylov
b330cb855f Fix build 2021-08-24 23:11:48 +00:00
alesapin
b274c2a90c
Update Dockerfile 2021-08-24 17:09:10 +03:00
Nikita Mikhaylov
19fc723328 chmod +x 2021-08-24 11:14:37 +00:00
Vitaly Baranov
8dc128f3d9 Set version of tzlocal to 2.1 2021-08-24 14:07:48 +03:00
Nikolay Degterinsky
d60f128a9a Merge branch 'master' into ncb/import-export-lz4 2021-08-24 11:47:10 +03:00
kssenii
b536653851 Merge branch 'master' of https://github.com/ClickHouse/ClickHouse into disk-over-web-server 2021-08-24 07:29:50 +00:00
Vladimir C
98acccbdcc
Merge branch 'master' into s3partitionedwrite 2021-08-24 10:03:57 +03:00