Commit Graph

56852 Commits

Author SHA1 Message Date
Alexander Kuzmenkov
a164abf23e add frame fuzzing and one query that fails under msan 2021-02-03 09:42:54 +03:00
Alexander Kuzmenkov
7e945bab03 fix the calculation for moving frame start 2021-02-03 08:53:21 +03:00
alexey-milovidov
4cbd0ed282
Merge pull request #20010 from ClickHouse/coverity
Fix some of the issues found by Coverity
2021-02-03 08:48:05 +03:00
alexey-milovidov
a4d86c43e9
Merge pull request #19946 from ClickHouse/improve-performance-of-aggregate-functions
Improve performance of aggregate functions
2021-02-03 08:46:51 +03:00
Alexander Kuzmenkov
d0bac8bfd9 Merge remote-tracking branch 'origin/master' into HEAD 2021-02-03 08:45:16 +03:00
alexey-milovidov
f9a11bb720
Merge pull request #19971 from azat/flaky-tests
Fix flaky integration tests
2021-02-03 08:42:45 +03:00
Alexander Kuzmenkov
85f8918753 fix the row order to be like in the original test 2021-02-03 07:47:53 +03:00
bharatnc
73fe3264af Fix spacing 2021-02-02 18:26:36 -08:00
bharatnc
c60c0664b7 Docs - add minor note about htt_interserver_credentials 2021-02-02 18:23:32 -08:00
bharatnc
76cd42f485 Docs - HTTPS interserver replication 2021-02-02 18:15:31 -08:00
Aleksei Semiglazov
d05c6446b9
Send cancel packet and cancel read_context before retrying the query 2021-02-03 00:07:00 +00:00
filimonov
5e350bb761
Update entrypoint.sh
Fix for #20002
2021-02-02 23:36:09 +01:00
Azat Khuzhin
fabace394d Cleanup message brokers output streams 2021-02-03 00:32:36 +03:00
Alexey Milovidov
191389d040 Fix some of the issues found by Coverity 2021-02-02 22:08:32 +03:00
Alexey Milovidov
12485eee6b Fix some of the issues found by Coverity 2021-02-02 22:07:23 +03:00
Azat Khuzhin
ac510f9770 test_read_temporary_tables_on_failure: increase timeout to avoid false-positive
In [1]:

    start_cluster = <helpers.cluster.ClickHouseCluster object at 0x7f2d46895dd8>

        def test_different_versions(start_cluster):
            with pytest.raises(QueryTimeoutExceedException):
                node.query("SELECT sleep(3)", timeout=1)
            with pytest.raises(QueryRuntimeException):
    >           node.query("SELECT 1", settings={'max_concurrent_queries_for_user': 1})
    E           Failed: DID NOT RAISE <class 'helpers.client.QueryRuntimeException'>

  [1]: https://clickhouse-test-reports.s3.yandex.net/19451/b68508002d134ead05bf2ca0e22a13a34a5c55c6/integration_tests_(thread).html#fail1
2021-02-02 22:04:45 +03:00
Azat Khuzhin
33628e1bd9 test_materialize_mysql_database: add retries for test_select_without_columns_8_0
Here is an example:

    2021.01.31 14:04:43.615618 [ 54 ] {} <Debug> executeQuery: (internal) /*Materialize MySQL step 1: execute dump data*/ INSERT INTO t(a, b, _sign, _version) VALUES
    2021.01.31 14:04:43.616453 [ 54 ] {} <Trace> db.t (fc55a47e-1cae-4f41-9a13-91ccde5a73bb): Renaming temporary part tmp_insert_0_1_1_0 to 0_1_1_0.
    ...
    2021.01.31 14:04:43.786823 [ 12 ] {4a0d1f80-4a96-4809-a84e-d55d7cd25cdd} <Debug> executeQuery: (from 172.18.0.1:45162, using production parser) SELECT count(_version) FROM db.t FORMAT TSV
    ...
    2021.01.31 14:04:44.566037 [ 54 ] {} <Debug> executeQuery: (internal) /*Materialize MySQL step 1: execute dump data*/ INSERT INTO t(a, b, _sign, _version) VALUES
    2021.01.31 14:04:44.566943 [ 54 ] {} <Trace> db.t (fc55a47e-1cae-4f41-9a13-91ccde5a73bb): Renaming temporary part tmp_insert_0_2_2_0 to 0_2_2_0.
    ...
    2021.01.31 14:04:52.343911 [ 54 ] {} <Error> MaterializeMySQLSyncThread: Code: 1002, e.displayText() = DB::Exception: Master maybe lost, Stack trace (when copying this message, always include the lines below):

  [1]: https://clickhouse-test-reports.s3.yandex.net/19584/978f23e3437d0d0eb64727e87ffe1b80289c9fc2/integration_tests_(release).html#fail1
2021-02-02 22:04:45 +03:00
Azat Khuzhin
3d3d6777d3 test_dictionaries_dependency_xml: decrease dictionaries lifetime
Since ExternalLoader::PeriodicUpdater::check_period_sec = 5, and so if
it will be scheduled too early the reload will be skipped, and indeed
this is what you can see in logs [1], the reload is done each 10
seconds, not 5:

    2021.01.31 14:20:22.590999 [ 48 ] {} <Trace> ExternalDictionariesLoader: Supposed update time for 'dep_x' is 2021-01-31 14:20:27 (loaded, lifetime [5, 5], no errors)
    2021.01.31 14:20:22.591016 [ 48 ] {} <Trace> ExternalDictionariesLoader: Next update time for 'dep_x' was set to 2021-01-31 14:20:27
    ...
    2021.01.31 14:20:32.164882 [ 50 ] {} <Trace> ExternalDictionariesLoader: Start loading object 'dep_x'

  [1]: https://clickhouse-test-reports.s3.yandex.net/19584/37797fdf5b30dc97147e73b3ac8ca9025b80aaed/integration_tests_(release).html#fail1
2021-02-02 22:04:32 +03:00
Aleksei Semiglazov
921518db0a CLICKHOUSE-606: query deduplication based on parts' UUID
* add the query data deduplication excluding duplicated parts in MergeTree family engines.

query deduplication is based on parts' UUID which should be enabled first with merge_tree setting
assign_part_uuids=1

allow_experimental_query_deduplication setting is to enable part deduplication, default ot false.

data part UUID is a mechanism of giving a data part a unique identifier.
Having UUID and deduplication mechanism provides a potential of moving parts
between shards preserving data consistency on a read path:
duplicated UUIDs will cause root executor to retry query against on of the replica explicitly
asking to exclude encountered duplicated fingerprints during a distributed query execution.

NOTE: this implementation don't provide any knobs to lock part and hence its UUID. Any mutations/merge will
update part's UUID.

* add _part_uuid virtual column, allowing to use UUIDs in predicates.

Signed-off-by: Aleksei Semiglazov <asemiglazov@cloudflare.com>

address comments
2021-02-02 16:53:39 +00:00
keenwolf
b7e4f7bd1e modify file attribute 2021-02-03 00:41:11 +08:00
Alexander Kuzmenkov
264aea20be Merge remote-tracking branch 'origin/master' into HEAD 2021-02-02 19:12:46 +03:00
keenwolf
9beb7689cc add comments and update test cases. 2021-02-02 23:35:45 +08:00
Vitaly Baranov
9427d5d94b
Merge pull request #19805 from vitlibar/allow-getting-return-type-of-dictget-without-loading-dictionary
Fix starting the server with tables having default expressions containing dictGet()
2021-02-02 18:19:55 +03:00
Alexander Kuzmenkov
d2a0e44a3a
Merge pull request #19299 from ClickHouse/aku/window-prototype
[wip] support RANGE frame for window functions
2021-02-02 17:59:16 +03:00
Alexander Kuzmenkov
6317208add
Merge branch 'master' into aku/perf 2021-02-02 17:24:17 +03:00
Alexey Milovidov
b9647e5326 Fix double whitespace 2021-02-02 17:22:03 +03:00
Alexander Kuzmenkov
b4afc49d3c update perf tests
sed -i s'/^<test.*$/<test>/g' tests/performance/*.xml

WITH ceil(max(q[3]), 1) AS h
SELECT concat('sed -i s\'/^<test.*$/<test max_ignored_relative_change="', toString(h), '">/g\' tests/performance/', test, '.xml') AS s
FROM
(
    SELECT
        test,
        query_index,
        count(*),
        min(event_time),
        max(event_time) AS t,
        arrayMap(x -> floor(x, 3), quantiles(0, 0.5, 0.95, 1)(stat_threshold)) AS q,
        median(stat_threshold) AS m
    FROM perftest.query_metrics_v2
    WHERE ((pr_number != 0) AND (event_date > '2021-01-01')) AND (metric = 'client_time') AND (abs(diff) < 0.05) AND (old_value > 0.1)
    GROUP BY
        test,
        query_index,
        query_display_name
    HAVING (t > '2021-01-01 00:00:00') AND ((q[3]) > 0.1)
    ORDER BY test DESC
)
GROUP BY test
ORDER BY h DESC
FORMAT PrettySpace
2021-02-02 17:21:43 +03:00
alexey-milovidov
695e28079d
Merge pull request #19960 from ClickHouse/json-extract-inaccurate-conversion-from-double-to-float
Allow inaccurate conversion from double to float in function JSONExtract beacuse the users want that
2021-02-02 15:15:37 +03:00
Anton Popov
fb02d56534
Merge pull request #19519 from olgarev/revolg-DOCSUP-4962-WHERE_and_GROUP_BY_in_TTL_expressions
DOCSUP-4962: Documented support of WHERE and GROUP BY in TTL expressions
2021-02-02 14:39:45 +03:00
alexey-milovidov
4d0973cf25
Merge pull request #19959 from depressed-pho/fix-segv-fromModifiedJulianDay
Fix a segfault in fromModifiedJulianDay()
2021-02-02 13:50:14 +03:00
Ilya Yatsishin
c5be9e3a0e
Merge pull request #19686 from benbiti/master
[Docs]Translation: update FINAL with max_final_threads setting to CN
2021-02-02 13:06:45 +03:00
alexey-milovidov
05b8164133
Merge pull request #19956 from ClickHouse/geohash-ubsan
Fix UBSan report in geoHashesInBox
2021-02-02 12:52:01 +03:00
Maksim Kita
c889f51e9e Updated performance benchmark 2021-02-02 12:50:58 +03:00
alexey-milovidov
8f90367e99
Merge pull request #19957 from ClickHouse/msan-system-stack-trace
Add MSan annotation for system.stack_trace
2021-02-02 12:50:52 +03:00
Alexey Milovidov
c4f3f900ec Merge branch 'master' into improve-performance-of-aggregate-functions 2021-02-02 12:11:22 +03:00
Alexey Milovidov
0b166e1cbd Add performance test 2021-02-02 12:11:20 +03:00
Yatsishin Ilya
19d6e867c6 HDFS test had a clash with 9000 ClickHouse port. 2021-02-02 12:08:47 +03:00
Kruglov Pavel
fd5adf1ff6
Merge pull request #19925 from ClickHouse/fix_insert_values_with_semicolon
Fix logical error in INSERT VALUES
2021-02-02 12:05:19 +03:00
tavplubix
befee42ab0
Merge pull request #19684 from yiguolei/master
Should fail ddl query as soon as possible if table is shutdown
2021-02-02 11:46:58 +03:00
tavplubix
dbcddbcebd
Merge pull request #19770 from stigsb/materializemysql_system_parts
Show MaterializeMySQL tables in system.parts
2021-02-02 11:43:42 +03:00
alexey-milovidov
f5dc38a33c
Merge pull request #19584 from azat/client-complete-error-codes
client: more suggestions
2021-02-02 11:05:17 +03:00
Maksim Kita
d0151de4bb
Merge pull request #19608 from kreuzerkrieg/Add_IStoragePolicy_interface
Add IStoragePolicy interface
2021-02-02 11:03:20 +03:00
alesapin
a9c50caa9a
Merge pull request #19942 from ClickHouse/fix_test_keeper_b2b_test
Fix test keeper integration tests
2021-02-02 10:17:27 +03:00
alexey-milovidov
da367a500e
Merge pull request #19963 from ClickHouse/adopters-fix-markup
Fix extremely inconvenient Markdown markup in Adopters page
2021-02-02 09:05:23 +03:00
Alexey Milovidov
3bc7a64abe Fix extremely inconvenient Markdown markup in Adopters page 2021-02-02 08:27:07 +03:00
keenwolf
f21e22ae2c add test for function timezoneOffset 2021-02-02 12:22:45 +08:00
Alexey Milovidov
078011ef2d Allow conversion from double to float in function JSONExtract beacuse the users want that 2021-02-02 07:16:47 +03:00
alexey-milovidov
8c0ec5105b
Add a patch from @FishermanZzhang, #19952 2021-02-02 06:53:11 +03:00
Alexey Milovidov
06bd0345eb Merge branch 'master' into improve-performance-of-aggregate-functions 2021-02-02 06:48:08 +03:00
Alexey Milovidov
656cd583f7 Add MSan annotation for system.stack_trace 2021-02-02 06:46:54 +03:00