Commit Graph

189 Commits

Author SHA1 Message Date
Alexander Tokmakov
7b1b238d0b Revert "Merge pull request #25674 from amosbird/distributedreturnconnection"
This reverts commit 5ffd99dfd4, reversing
changes made to 2796aa333f.
2023-03-11 19:09:47 +01:00
Clayton McClure
b0931c8967
Update copier to use group by to find partitions (#47386) 2023-03-09 15:49:00 +01:00
Antonio Andelic
7f5fb77ed5
Increase table retries in cluster copier tests (#46590) 2023-02-22 15:09:48 +01:00
Antonio Andelic
8cdafcc2d0 Correctly check if there were failures 2023-02-15 11:45:57 +00:00
Antonio Andelic
8d16fe5793
Merge branch 'master' into add-support-for-settings-alias 2023-02-13 08:46:00 +01:00
Antonio Andelic
f96d480563
Merge branch 'master' into add-support-for-settings-alias 2023-02-09 16:07:45 +01:00
Robert Schulze
e490ec91d9
Merge branch 'master' into rs/fix-fragile-linking 2023-02-09 11:33:59 +01:00
Alexander Gololobov
a7c30eecf4
Merge pull request #46141 from ClickHouse/fix-assertion-in-statistics
Fix assertion in statistical functions
2023-02-08 13:42:02 +01:00
Robert Schulze
6ff232d782
Merge branch 'master' into rs/fix-fragile-linking 2023-02-08 12:51:12 +01:00
Alexey Milovidov
55c3bbb739 Fix assertion in statistical functions 2023-02-08 00:09:41 +01:00
Azat Khuzhin
ff324fe81a Fix fault injection in copier (wrong query syntax)
There are very frequent flakiness of `test_cluster_copier` test, here is
an example of copier failures on CI [1]:

    AssertionError: Instance: s0_1_0 (172.16.29.9). Info: {'ID': '5d68dcb46fdb4b0c54b7c7ba1ddde83b8f34d483bbb32abcb0c52b966444ce82', 'Running': False, 'ExitCode': 85, 'ProcessConfig': {'tty': False, 'entrypoint': '/usr/bin/clickhouse', 'arguments': ['copier', '--config', '/etc/clickhouse-server/config-copier.xml', '--task-path', '/clickhouse-copier/task_simple_4DFWYTDD49', '--task-file', '/task0_description.xml', '--task-upload-force', 'true', '--base-dir', '/var/log/clickhouse-server/copier', '--copy-fault-probability', '0.2', '--experimental-use-sample-offset', '1'], 'privileged': False, 'user': '0'}, 'OpenStdin': False, 'OpenStderr': True, 'OpenStdout': True, 'CanRemove': False, 'ContainerID': 'f356df6694b3cc09ee9830c623681626f8e8d999677c188b9fe911aa702784ca', 'DetachKeys': '', 'Pid': 84332}
    assert 85 == 0

But let's look what the error it is, apparently it is UNFINISHED:

    SELECT
        name,
        code
    FROM system.errors
    WHERE ((code % 256) = 85) AND (NOT remote)
    SETTINGS system_events_show_zero_values = 1

    ┌─name─────────────────────────────┬─code─┐
    │ FORMAT_IS_NOT_SUITABLE_FOR_INPUT │   85 │
    │ UNFINISHED                       │  341 │
    │ NO_SUCH_ERROR_CODE               │  597 │
    └──────────────────────────────────┴──────┘

Let's verify:

    $ grep -r UNFINISHED ./test_cluster_copier/_instances_0/s0_1_0/logs/copier/clickhouse-copier_*
    ./test_cluster_copier/_instances_0/s0_1_0/logs/copier/clickhouse-copier_20230206220846_368/log.log:2023.02.06 22:09:19.015251 [ 368 ] {} <Error> : virtual int DB::ClusterCopierApp::main(const std::vector<std::string> &): Code: 341. DB::Exception: Too many tries to process table cluster1.default.hits. Abort remaining execution. (UNFINISHED), Stack trace (when copying this message, always include the lines below):

And apparently that it is due to query error with fault injection:

    2023.02.06 22:09:15.654724 [ 368 ] {} <Error> Application: An error occurred while processing partition 0: Code: 62. DB::Exception: Syntax error (Query): failed at position 168 ('Native'): Native. Expected one of: token, Dot, OR, AND, BETWEEN, NOT BETWEEN, LIKE, ILIKE, NOT LIKE, NOT ILIKE, IN, NOT IN, GLOBAL IN, GLOBAL NOT IN, MOD, DIV, IS NULL, IS NOT NULL, alias, AS, Comma, OFFSET, WITH TIES, BY, LIMIT, SETTINGS, UNION, EXCEPT, INTERSECT, INTO OUTFILE, FORMAT, end of query. (SYNTAX_ERROR), Stack trace (when copying this message, always include the lines below):

Example:

    select x from x limit  1FORMAT Native

    Syntax error: failed at position 32 ('Native'):

So fixing this should fix test_cluster_copier flakiness.

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/46045/bd4170e03c6af583a51d12d2c39fa775dcb9997b/integration_tests__release__[4/4].html

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-02-07 17:01:06 +01:00
Azat Khuzhin
18d2a99761 Use comments for variables that clang can parse
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-02-07 16:53:45 +01:00
Robert Schulze
84b9ff450f
Fix terribly broken, fragile and potentially cyclic linking
Sorry for the clickbaity title. This is about static method
ConnectionTimeouts::getHTTPTimeouts(). It was be declared in header
IO/ConnectionTimeouts.h, and defined in header
IO/ConnectionTimeoutsContext.h (!). This is weird and caused issues with
linking on s390x (##45520). There was an attempt to fix some
inconsistencies (#45848) but neither did @Algunenano nor me at first
really understand why the definition is in the header.

Turns out that ConnectionTimeoutsContext.h is only #include'd from
source files which are part of the normal server build BUT NOT part of
the keeper standalone build (which must be enabled via CMake
-DBUILD_STANDALONE_KEEPER=1). This dependency was not documented and as
a result, some misguided workarounds were introduced earlier, e.g.
0341c6c54b

The deeper cause was that getHTTPTimeouts() is passed a "Context". This
class is part of the "dbms" libary which is deliberately not linked by
the standalone build of clickhouse-keeper. The context is only used to
read the settings and the "Settings" class is part of the
clickhouse_common library which is linked by clickhouse-keeper already.

To resolve this mess, this PR

- creates source file IO/ConnectionTimeouts.cpp and moves all
  ConnectionTimeouts definitions into it, including getHTTPTimeouts().

- breaks the wrong dependency by passing "Settings" instead of "Context"
  into getHTTPTimeouts().

- resolves the previous hacks
2023-02-05 20:49:34 +00:00
Antonio Andelic
8929838fcc Merge branch 'master' into add-support-for-settings-alias 2023-02-03 17:47:00 +00:00
Nikita Mikhaylov
33877b5e00
Parallel replicas. Part [2] (#43772) 2023-02-03 14:34:18 +01:00
Antonio Andelic
f613bfd8d2 Merge branch 'master' into add-support-for-settings-alias 2023-02-02 11:15:39 +00:00
Raúl Marín
7c31cb7adc Proper includes for ConnectionTimeoutsContext.h 2023-01-31 16:11:32 +01:00
Antonio Andelic
714fad1529 Add support for settings alias 2023-01-26 14:06:46 +00:00
Alexander Tokmakov
70d1adfe4b
Better formatting for exception messages (#45449)
* save format string for NetException

* format exceptions

* format exceptions 2

* format exceptions 3

* format exceptions 4

* format exceptions 5

* format exceptions 6

* fix

* format exceptions 7

* format exceptions 8

* Update MergeTreeIndexGin.cpp

* Update AggregateFunctionMap.cpp

* Update AggregateFunctionMap.cpp

* fix
2023-01-24 00:13:58 +03:00
Kruglov Pavel
f6f71ac5f1
Merge branch 'master' into fix_bug_in_copier 2022-12-01 12:29:50 +01:00
Alexey Milovidov
cf27a0cdeb Fix log messages in clickhouse-copier 2022-11-25 22:29:02 +01:00
lzydmxy
0462132dd6 Ensure consistency when copier update status and current_partition_attach_is_done after partition attach is done 2022-11-24 11:15:44 +08:00
Azat Khuzhin
44f23c2568 Make disks checks only for clickhouse-server
This will fix clickhouse-disks

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-11-20 16:28:35 +01:00
Vitaly Baranov
8d72f75556 Make the password wiping an option of IAST::format(). 2022-11-16 01:35:06 +01:00
Robert Schulze
e996e2ff15
Fix clang-tidy 2022-10-26 07:48:37 +00:00
Robert Schulze
c119cd2f00
Merge branch 'master' into update-libcxx-to-15 2022-10-24 08:29:37 +02:00
Azat Khuzhin
4e76629aaf Fixes for -Wshorten-64-to-32
- lots of static_cast
- add safe_cast
- types adjustments
  - config
  - IStorage::read/watch
  - ...
- some TODO's (to convert types in future)

P.S. That was quite a journey...

v2: fixes after rebase
v3: fix conflicts after #42308 merged
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2022-10-21 13:25:19 +02:00
Robert Schulze
f435e4fbe2
Un-clusterf$ck incomplete type errors 2022-10-20 14:48:46 +00:00
Robert Schulze
50c932e566
Separate TaskShard and TaskTable 2022-10-20 14:00:20 +00:00
Robert Schulze
df318d8719
Un-inline TaskTableAndShard 2022-10-20 13:43:15 +00:00
Robert Schulze
f6de964eaa
Fix static_assert in random generator 2022-10-20 11:20:18 +00:00
Robert Schulze
cdae7082a3
Up-inline ShardPartition, ShardPartitionPiece and TaskCluster 2022-10-20 11:16:05 +00:00
Alexey Milovidov
35cce03125 Remove dlopen 2022-09-17 03:02:34 +02:00
Yakov Olkhovskiy
2e34b384c1 update tcp protocol, add quota_key 2022-08-03 15:44:08 -04:00
Robert Schulze
62149111d3
Fix build 2022-07-14 14:36:31 +00:00
Nikolai Kochetov
6e924cdc77 Fix some more tests. 2022-06-01 15:21:47 +00:00
Nikolai Kochetov
3d84aae0ab Better. 2022-05-24 20:06:08 +00:00
Nikolai Kochetov
333fd09dbf Fixing build. 2022-05-24 19:29:00 +00:00
Amos Bird
4a5e4274f0
base should not depend on Common 2022-04-29 10:26:35 +08:00
Robert Schulze
118e94523c
Activate clang-tidy warning "readability-container-contains"
This check suggests replacing <Container>.count() by
<Container>.contains() which is more speaking and in case of
multimaps/multisets also faster.
2022-04-18 23:53:11 +02:00
vdimir
e7d3dbeebd
cluster discovery init only with zookeeper 2021-12-27 15:26:08 +03:00
Raúl Marín
b2cfa70541 Reduce dependencies on ASTFunction.h
481 -> 230
2021-11-26 18:21:54 +01:00
Raúl Marín
051dddd8df Reduce dependencies on ASTIdentifier.h
Goes from rebuilding 483 objects to 165 when it's modified
2021-11-26 16:49:40 +01:00
Nikolay Degterinsky
ba50e8387d Merge branch 'master' into query_parameters 2021-11-15 19:55:54 +00:00
Azat Khuzhin
48e9456e75 Disable log_queries in clickhouse-copier
This will avoid calling Context::addQueryFactoriesInfo()

v2: disable it in ClusterCopier::reloadTaskDescription() too
v3: just create local context and pass it to the ClusterCopier instead
of global context
2021-11-15 10:32:30 +03:00
Azat Khuzhin
6274c7bfc2 Do not use global context for query execution in clickhouse-copier 2021-11-15 10:32:19 +03:00
Nikolay Degterinsky
d76976e272 Merge branch 'master' into query_parameters 2021-10-18 17:34:20 +03:00
Nikolai Kochetov
a08c98d760 Move some files. 2021-10-16 17:03:50 +03:00
Nikolai Kochetov
067eaadadd Merge branch 'master' into removing-data-streams-folder 2021-10-16 09:46:05 +03:00
Nikolai Kochetov
c668696047
Merge pull request #30171 from ClickHouse/remove-stream-interfaces
Remove stream interfaces
2021-10-16 09:34:01 +03:00