Commit Graph

78404 Commits

Author SHA1 Message Date
Azat Khuzhin
3e96b28843 perf: convert killall to pkill (since killall has some magic, see -e option) 2021-12-03 10:56:42 +03:00
Azat Khuzhin
fb0cc62509 perf: fix waiting of the server after running tests
killall requires strict match, i.e. "clickhouse-server" not
"clickhouse":

    2021-12-03 05:24:56      + env kill -- -21700
    2021-12-03 05:24:56      kill: (-21700): No such process
    2021-12-03 05:24:56      + killall clickhouse
    2021-12-03 05:24:56      clickhouse: no process found
    2021-12-03 05:24:56      + echo Servers stopped.
    2021-12-03 05:24:56      Servers stopped.
    2021-12-03 05:24:56      + analyze_queries

    $ tail -n1 *-server-log.log
    ==> left-server-log.log <==
    2021.12.03 05:26:59.530647 [ 450 ] {} <Trace> SystemLog (system.asynchronous_metric_log): Flushed system log up to offset 1668052

    ==> right-server-log.log <==
    2021.12.03 05:27:20.873136 [ 466 ] {} <Trace> SystemLog (system.metric_log): Flushed system log up to offset 9605

    ==> setup-server-log.log <==
    2021.12.03 02:47:14.844395 [ 96 ] {} <Information> Application: Child process exited normally with code 0.

As you can see killall instantly fails with no such process, while this
cannot be true since it was there, and also according to logs there were
messages after running analyze_queries() from compare.sh

This should fix problems like in [1].

  [1]: https://clickhouse-test-reports.s3.yandex.net/32080/344298f4037f88b114b8e798bb30036b24be8f16/performance_comparison/report.html#fail1
2021-12-03 10:55:35 +03:00
alesapin
7699d3f35e
Merge pull request #32162 from azat/double-gz
tests/ci: do not compress logs twice
2021-12-03 10:20:35 +03:00
alesapin
83c086b051
Merge pull request #32120 from ClickHouse/add_one_more_group
Move fuzzers and unit tests to another group
2021-12-03 10:14:09 +03:00
Kseniia Sumarokova
03c7907b37
Merge pull request #32172 from ClickHouse/kssenii-patch-2
Update materialized-postgresql.md
2021-12-03 09:28:46 +03:00
Kseniia Sumarokova
9093ad6596
Update materialized-postgresql.md 2021-12-03 09:27:22 +03:00
Kseniia Sumarokova
a8dcbc988a
Merge pull request #32129 from kssenii/fix-postgres-tests
fixing postgres tests
2021-12-03 09:15:20 +03:00
alexey-milovidov
b7fb21e6f0
Merge pull request #32163 from azat/clickhouse-test-random
clickhouse-test: do not use random generator with shared state
2021-12-03 06:53:57 +03:00
alexey-milovidov
17fcf0a6d3
Merge pull request #32165 from azat/query-profiler-reset-fix
Fix QueryProfiler (query_profiler_{cpu,real}_time_period_ns) reset
2021-12-03 06:51:21 +03:00
Anton Popov
f6be3d16fd
Merge pull request #24820 from kssenii/versioning
Versioning of aggregate function states
2021-12-03 01:41:44 +03:00
Azat Khuzhin
0b2de32228 Fix QueryProfiler (query_profiler_{cpu,real}_time_period_ns) reset
Even after timer_delete() the signal can be fired.
Reproducer:

    $ clickhouse-server & # with configured trace_log
    $ clickhouse-benchmark -c2 --query 'select * from numbers(1e6)' --query_profiler_cpu_time_period_ns=1 &
    ...
    2021.12.02 14:28:01.320288 [ 24885 ] {} <Debug> TCPHandler: Processed in 177.055205644 sec.
    User defined signal 2

CI failures:
- https://s3.amazonaws.com/clickhouse-test-reports/32067/8dbc7a8dae17090a18778f29629d8746a1bb9b72/stateful_tests__debug__actions_.html
- https://s3.amazonaws.com/clickhouse-test-reports/32064/c07450a7dce363b7a4c5ca3ab0e833c25e3d46c0/stateful_tests__debug__actions_.html

Fix this by do not reset the signal back, and introduce a flag to ignore
signals after disabling the timer.

Fixes: #31740
2021-12-03 01:33:08 +03:00
Azat Khuzhin
96bd83c31e Do not reopen logs on USR1, HUP is enough
USR1 is also used for query_profiler_real_time_period_ns, let's not
overlap.
2021-12-03 01:32:57 +03:00
Azat Khuzhin
9276384977 clickhouse-test: do not use random generator with shared state
Recently (#32094) test database had been overlapped, and random prefix
for database had been increased from 6 to 8.

But actually 6 bytes for random prefix should be enough (with existing
alphabet (0-9a-z) it is 36**6=2'176'782'336), and the real reason of
this overlap is that random generator by default uses shared state [1]:

    The functions supplied by this module are actually bound methods of
    a hidden instance of the random.Random class. You can instantiate your
    own instances of Random to get generators that don’t share state.

  [1]: https://docs.python.org/3/library/random.html

I've played a little bit with random in python, and using default random
generator it generates non-unique strings pretty fast, just in a few
runs, but using SystemRandom (that uses /dev/urandom) it takes ~1 minute.

Test:

```sh
    $ while /tmp/test.py | LANG=c sort -S5G | LANG=c uniq -d | tee /dev/stderr | wc -l | fgrep -q -x -c 0; do :; done
```

```python
    #!/usr/bin/env python3

    import multiprocessing
    import string
    import random

    def random_str(length=6):
        alphabet = string.ascii_lowercase + string.digits
        return ''.join(random.SystemRandom().choice(alphabet) for _ in range(length))

    def worker(_):
        print(random_str())

    with multiprocessing.Pool(processes=2) as pool:
        pool.map(worker, range(0, int(10e3)))
```

So let's switch to SystemRandom and use 6-byte prefix.
2021-12-03 01:29:23 +03:00
Azat Khuzhin
bfaaa79cfc tests/ci: do not compress logs twice
Cc: @alesapin
2021-12-03 01:28:32 +03:00
Frank Chen
e895c85f49
Add exception/exception_code to trace span log (#32040) 2021-12-03 01:00:02 +03:00
kssenii
06312a2df9 Fix 2021-12-02 22:48:21 +03:00
mergify[bot]
e2cb91b560
Merge branch 'master' into add_one_more_group 2021-12-02 19:28:13 +00:00
alesapin
45021bd35c
Merge pull request #32121 from ClickHouse/add_assertion_in_keeper
Add check for duplicate hostnames and IDs in KeeperConfig
2021-12-02 22:24:46 +03:00
alesapin
308eadd83a
Merge pull request #32159 from ClickHouse/fix_ci
Fix CI
2021-12-02 22:04:48 +03:00
alesapin
f2140be90b Fix CI 2021-12-02 22:03:59 +03:00
tavplubix
4d96d6ab77
Merge pull request #32052 from stigsb/fix-flaky-create-table-like-test
Fix flaky integration test for MaterializedMySQL CREATE TABLE LIKE
2021-12-02 21:28:00 +03:00
alesapin
c14086d578
Merge pull request #32064 from ClickHouse/lightweight_checks_rerun
Add ability for lightweight checks rerun
2021-12-02 20:54:27 +03:00
alexey-milovidov
2a7a75ea88
Merge pull request #32156 from flickerbox/crb-update-featured-image
Update featured image for 21.11 release blog post
2021-12-02 20:35:48 +03:00
Cody Baker
be660ca25f Update featured image for 21.11 release blog post 2021-12-02 10:28:47 -07:00
alesapin
dd2ccd3b17
Merge pull request #32155 from ClickHouse/small_improvements
Small improvements in lambda code
2021-12-02 19:39:00 +03:00
alesapin
9e1437a7c4 Small improvements in lambda code 2021-12-02 19:38:18 +03:00
mergify[bot]
c399eaf945
Merge branch 'master' into add_one_more_group 2021-12-02 15:20:06 +00:00
kssenii
64fcbe347e Update 2021-12-02 16:08:11 +03:00
Nikolai Kochetov
7de34bb8e1
Merge pull request #32126 from ClickHouse/remove-02116_global_in_time_limit
Remove 02116_global_in_time_limit.
2021-12-02 16:07:18 +03:00
Nikolai Kochetov
4d10762727 Remove 02116_global_in_time_limit. 2021-12-02 16:05:06 +03:00
mergify[bot]
88e1a26e99
Merge branch 'master' into add_one_more_group 2021-12-02 12:56:57 +00:00
alesapin
b8bffcb14c
Merge pull request #32115 from ClickHouse/fix_flaky_keeper_tests
Fix flaky keeper whitelist test
2021-12-02 15:41:17 +03:00
tavplubix
1f4b14f68b
Merge pull request #32119 from ClickHouse/tavplubix-patch-4
Fix data race in `removePartAndEnqueueFetch(...)`
2021-12-02 15:25:38 +03:00
Kruglov Pavel
049b2c0c14
Merge pull request #31072 from MaxWk/feature-support-bool-type
Feature support bool type
2021-12-02 14:50:29 +03:00
alesapin
437591f2fa Add check for duplicate hostnames and IDs in KeeperConfig 2021-12-02 14:46:33 +03:00
tavplubix
5400c5899d
Merge pull request #32063 from ClickHouse/fix_detach_dir_exists
Fix 'directory exists' error when detaching part
2021-12-02 14:04:21 +03:00
Anton Popov
a79dbb6b6a
Merge pull request #32079 from Avogar/parse-plus-in-float
Allow plus before Float32/Float64
2021-12-02 14:03:48 +03:00
alesapin
ff6cc14df3 Fix one more test 2021-12-02 13:41:04 +03:00
alesapin
7ef0737edc Move fuzzers and unit tests to another group 2021-12-02 13:23:32 +03:00
tavplubix
e9489ccd6f
Update StorageReplicatedMergeTree.cpp 2021-12-02 13:18:02 +03:00
Maksim Kita
2d359a5e79
Merge pull request #32110 from bharatnc/ncb/h3-func-looping
make looping in H3 funcs uniform
2021-12-02 13:15:36 +03:00
alesapin
e704140d07 Fix flaky whitelist test 2021-12-02 12:37:08 +03:00
Maksim Kita
a426ed0a5a
Merge pull request #31800 from kitaisreal/dict-get-with-type-nullable-fix
Function dictGet with type Nullable fix
2021-12-02 12:27:47 +03:00
alesapin
1cda5bfe4e
Merge pull request #32114 from ClickHouse/remove_pvs_from_master
Remove PVS check from master
2021-12-02 11:37:00 +03:00
alesapin
97e7da18bb Remove PVS check from master 2021-12-02 11:36:30 +03:00
Maksim Kita
9874931347
Merge pull request #32095 from FedericoCeratto/patch-2
Fix syntax error: drop comma
2021-12-02 11:20:52 +03:00
kssenii
5a273d4dda Update version 2021-12-02 08:02:07 +00:00
mergify[bot]
cd4a58c59d
Merge branch 'master' into patch-2 2021-12-02 08:01:05 +00:00
alesapin
d273a8db7c
Merge pull request #32094 from azat/clickhouse-test-longer-db
Increase length of random database in clickhouse-test
2021-12-02 10:58:40 +03:00
kssenii
8edf146687 Merge branch 'master' of github.com:ClickHouse/ClickHouse into versioning 2021-12-02 07:57:14 +00:00