thevar1able/ClickHouse
1
0
Fork 0
mirror of https://github.com/ClickHouse/ClickHouse.git synced 2024-12-13 18:02:24 +00:00
Code Issues Packages Projects Releases Wiki Activity
107,643 Commits 9,328 Branches 1,783 Tags 2.1 GiB
1b207d9fa0
Commit Graph

11394 Commits

Author SHA1 Message Date
Sergei Trifonov
b0fc551c27
Merge branch 'master' into io-scheduler-integration 2023-02-08 18:16:00 +01:00
attack204
ecaa97c5b1 fix style 2023-02-08 23:45:39 +08:00
attack204
1328725054 fix style 2023-02-08 22:58:31 +08:00
attack204
01cac6c49b fix style 2023-02-08 22:49:10 +08:00
attack204
752d1f81b0 fix style 2023-02-08 22:36:07 +08:00
attack204
1fdf3659da fix style 2023-02-08 22:29:23 +08:00
attack204
41daf88d64 fix style 2023-02-08 22:20:47 +08:00
attack204
820e5bd9c0 fix style 2023-02-08 22:15:05 +08:00
attack204
7c6e30ebe2 fix pointer bug 2023-02-08 22:02:58 +08:00
attack204
b1fc12c162 delete extra API and use dynamic_cast to compute 2023-02-08 21:52:13 +08:00
Smita Kulkarni
87ec32f235 Updated to use znode_name instead of log_entry_id - Updated checking of SYSTEM SYNC REPLICA 2023-02-08 14:07:06 +01:00
attack204
b3e2f99f4f
Merge branch 'master' into addParts 2023-02-08 20:27:54 +08:00
attack204
9e1081b79a add parts, active_parts total_marks to system.tables 2023-02-08 20:23:06 +08:00
Robert Schulze
6ff232d782
Merge branch 'master' into rs/fix-fragile-linking 2023-02-08 12:51:12 +01:00
Azat Khuzhin
151ba92a2e Fix reading of non existing nested columns with multiple level in compact parts 
Consider the following example:

    CREATE TABLE data (root.array_str Array(UInt8)) ENGINE = MergeTree() ORDER BY tuple();
    INSERT INTO data VALUES ([]);
    ALTER TABLE data ADD COLUMN root.nested_array Array(Array(UInt8));

In this case the first part will not have data for root.nested_array,
and thanks to #37152 it will simply read offsets column from
root.array_str, however since root.nested_array is a nested array, it
will try to read elements from the same offsets stream and if you are
lucky enough you will get one of the following errors:

- Cannot read all data. Bytes read: 1. Bytes expected: 8.: (while reading column root.nested_array): While executing MergeTreeInOrder. (CANNOT_READ_ALL_DATA)
- DB::Exception: Array size is too large: 8233460228287709730: (while reading column serp.serp_features): While executing MergeTreeInOrder.

So to address this, findColumnForOffsets() had been changed to return
the level of the column too, to allow to read only up to this level.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-02-08 10:21:40 +01:00
Smita Kulkarni
8c9d994fd7 Extended scope of state_mutex while adding subscriber - Updated checking of SYSTEM SYNC REPLICA 2023-02-07 22:04:17 +01:00
Kseniia Sumarokova
41dbdcff67
Merge branch 'master' into simplify-storage-s3-configuration 2023-02-07 18:09:34 +01:00
Smita Kulkarni
ecea28a832 Fixed style check - Updated checking of SYSTEM SYNC REPLICA 2023-02-07 17:27:34 +01:00
Nikolai Kochetov
3912f5a333
Merge pull request #45681 from CurtizJ/compact-parts-vertical-merge 
Allow vertical merges from compact to wide parts
 2023-02-07 17:09:17 +01:00
Smita Kulkarni
34341ddabe Updated locks and removed getLogEntryIds function - Updated checking of SYSTEM SYNC REPLICA 2023-02-07 16:57:58 +01:00
kssenii
40deaf1661 Fix clang-tidy 2023-02-07 16:30:04 +01:00
Sema Checherinda
52d4e78b7f
Merge pull request #45907 from CheSema/forbid-wal-on-s3 
do not run wal on remote disks
 2023-02-07 14:48:00 +01:00
Yakov Olkhovskiy
4e7dffafe4
Merge pull request #46087 from ClickHouse/fix-ip-bloom 
Allow bloom filter for IPv4 and IPv6
 2023-02-07 08:29:28 -05:00
Robert Schulze
10af0b3e49
Reduce redundancies 2023-02-07 12:27:23 +00:00
kssenii
9485873a2f Fix integration test 2023-02-07 12:45:23 +01:00
Robert Schulze
08c1f8346e
Merge remote-tracking branch 'origin/master' into rs/fix-fragile-linking 2023-02-07 11:22:00 +00:00
Smita Kulkarni
fb765696b6 Removed state_mutex lock from addSubscriber - Updated checking of SYSTEM SYNC REPLICA 2023-02-07 12:19:58 +01:00
Robert Schulze
690d873126
Merge pull request #46088 from ClickHouse/rs/fix-liburing-build 
Fix build with `-DENABLE_LIBURING=0` (or `-DENABLE_LIBRARIES=0`)
 2023-02-07 12:16:14 +01:00
Sergei Trifonov
950c04cb0e
Merge pull request #44907 from azat/dist/async-INSERT-error-message-fix 
Fix error message for a broken distributed batches ("While sending batch")
 2023-02-07 12:14:52 +01:00
Vitaly Baranov
f021c0eab6
Merge pull request #46058 from vitlibar/remove-unnecessary-execute-for-const-expr 
Remove unnecessary execute() while evaluating a constant expression.
 2023-02-07 12:09:37 +01:00
Azat Khuzhin
8cc41b7f41 Check return value of ::close() 
Note, that according close(2), EINTR should not be retriable for close()

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-02-07 11:28:22 +01:00
Smita Kulkarni
135615cda6 Removed unwanted check from before callback - Updated checking of SYSTEM SYNC REPLICA 2023-02-07 10:27:45 +01:00
Smita Kulkarni
ef29b61786 Updated callback to return log entry ids - Updated checking of SYSTEM SYNC REPLICA 2023-02-07 10:02:38 +01:00
Smita Kulkarni
f34ef86332 Moved background task trigger before getting log entries to wait - Updated checking of SYSTEM SYNC REPLICA 2023-02-07 08:49:41 +01:00
Anton Popov
7dfea59267 remove an assertion 2023-02-07 00:50:41 +00:00
Alexander Tokmakov
13da43ead8
Merge branch 'master' into fix_test_00002 2023-02-07 03:25:42 +03:00
kssenii
6cea67d58b Cleanup 2023-02-06 18:11:33 +01:00
Robert Schulze
8840aa9242
Fix build with -DENABLE_LIBURING=0 (or -DENABLE_LIBRARIES=0) 
Follow-up to #38456

The problem was that code which uses liburing was conditionally build
depending on define OS_LINUX and not based on whether liburing was
enabled/disabled at build time.
 2023-02-06 16:57:42 +00:00
Yakov Olkhovskiy
e433c26cf9 allow bloom filter for IPv4 and IPv6 2023-02-06 16:30:18 +00:00
Alexander Tokmakov
af655eb7dc
Merge pull request #45889 from CheSema/release-merge-blocker-drop-part 
do not block merges when old parts are droping in drop queries
 2023-02-06 19:09:09 +03:00
kssenii
ab0dedf0c8 Simplify code around storage s3 configuration 2023-02-06 16:23:17 +01:00
Alexander Tokmakov
98a4e1ab2c better comment 2023-02-06 15:08:30 +01:00
Alexander Tokmakov
86feaae253 Merge branch 'master' into explicit_drop_part_entry_type 2023-02-06 14:59:59 +01:00
Alexander Tokmakov
e9f770d89d fix 2023-02-06 14:03:14 +01:00
alesapin
b94f9f8006
Merge branch 'master' into 45508_Update_strategy_for_system_sync_replica 2023-02-06 12:38:39 +01:00
Kseniia Sumarokova
38c001ca42
Merge pull request #45957 from xiedeyantu/s3_file_not_found 
Throw an error on no files satisfying S3 wildcard
 2023-02-06 12:32:12 +01:00
Alexander Gololobov
d18364476d
Merge pull request #46047 from azat/system.processes-elapsed-fix 
Fix elapsed column in system.processes (10x error)
 2023-02-06 12:26:05 +01:00
Sema Checherinda
734d3a318b
const auto ref in the loop 2023-02-06 12:09:33 +01:00
Robert Schulze
84b9ff450f
Fix terribly broken, fragile and potentially cyclic linking 
Sorry for the clickbaity title. This is about static method
ConnectionTimeouts::getHTTPTimeouts(). It was be declared in header
IO/ConnectionTimeouts.h, and defined in header
IO/ConnectionTimeoutsContext.h (!). This is weird and caused issues with
linking on s390x (##45520). There was an attempt to fix some
inconsistencies (#45848) but neither did @Algunenano nor me at first
really understand why the definition is in the header.

Turns out that ConnectionTimeoutsContext.h is only #include'd from
source files which are part of the normal server build BUT NOT part of
the keeper standalone build (which must be enabled via CMake
-DBUILD_STANDALONE_KEEPER=1). This dependency was not documented and as
a result, some misguided workarounds were introduced earlier, e.g.
0341c6c54b

The deeper cause was that getHTTPTimeouts() is passed a "Context". This
class is part of the "dbms" libary which is deliberately not linked by
the standalone build of clickhouse-keeper. The context is only used to
read the settings and the "Settings" class is part of the
clickhouse_common library which is linked by clickhouse-keeper already.

To resolve this mess, this PR

- creates source file IO/ConnectionTimeouts.cpp and moves all
  ConnectionTimeouts definitions into it, including getHTTPTimeouts().

- breaks the wrong dependency by passing "Settings" instead of "Context"
  into getHTTPTimeouts().

- resolves the previous hacks
 2023-02-05 20:49:34 +00:00
Vitaly Baranov
a3423ce196 Remove unnecessary execute() while evaluating a constant expression. 2023-02-05 20:26:34 +01:00
Han Fei
532b341de9
Merge pull request #45975 from ucasfl/_part 
use LowCardnality for _part and _partition_id virtual column
 2023-02-05 18:00:46 +01:00
kssenii
a7eb1b1419 Fix integration test 2023-02-04 22:57:30 +01:00
Azat Khuzhin
1307e6d323 Fix elapsed column in system.processes (10x error) 
Fixes: #43455 (cc @Algunenano)
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-02-04 22:37:11 +01:00
kssenii
5abbc7cad6 Cleanup 2023-02-04 19:41:41 +01:00
kssenii
1cbff88a7c Fix integration test 2023-02-04 19:31:19 +01:00
xiedeyantu
f13eedd644 change settings name 2023-02-04 22:11:14 +08:00
Alexey Milovidov
22bd0b6f69
Merge pull request #38983 from CurtizJ/randomize-mt-settings 
Allow to randomize merge tree settings in tests
 2023-02-04 02:59:52 +01:00
kssenii
5f124263a6 Revert changes with complex SettingsChanges 2023-02-03 21:04:45 +01:00
Alexander Tokmakov
3f11948bb0
Merge branch 'master' into stack_trace_in_part_log 2023-02-03 20:05:00 +03:00
Alexander Tokmakov
7e6f7c79f2
Merge pull request #45457 from FrankChen021/exception_time 
Add last_exception_time to replication_queue
 2023-02-03 20:00:15 +03:00
Nikita Mikhaylov
33877b5e00
Parallel replicas. Part [2] (#43772) 2023-02-03 14:34:18 +01:00
Antonio Andelic
d5117f2aa6
Define S3 client with bucket and endpoint resolution (#45783) 
* Update aws

* Define S3 client with bucket and endpoint resolution

* Add defines for ErrorCodes

* Use S3Client everywhere

* Remove unused errorcode

* Add DROP S3 CLIENT CACHE query

* Add a comment

* Fix style

* Update aws

* Update reference files

* Add missing include

* Fix unit test

* Remove unneeded declarations

* Correctly use RetryStrategy

* Rename S3Client to Client

* Fix retry count

* fix clang-tidy warnings
 2023-02-03 14:30:52 +01:00
Vitaly Baranov
45d2d678ab
Merge pull request #45800 from vitlibar/rename-new-columns-in-system-backups 
Rename new columns in system.backups
 2023-02-03 14:00:16 +01:00
Azat Khuzhin
a196f995b1 Fix error message for a broken distributed batches ("While sending batch") 
There was an error from the begginning that does not respect
file_indices, and iterate only over file_index_to_path, while this is
not correct, since there can be less files then in file_index_to_path,
and this is what file_indices for.

Note, that only an error message was wrong, logic was fine. You can
verify this by the logs:

    2022.12.07 11:55:50.951976 [ 39217 ] {} <Debug> default.dist.DirectoryMonitor: Sending a batch of 10 files to localhost:9000 (128.42 thousand rows, 36.32 MiB bytes).
    2022.12.07 11:55:50.953762 [ 39217 ] {} <Error> default.dist.DirectoryMonitor: Code: 516. DB::Exception: Received from localhost:9000. DB::Exception: Interserver authentication failed. Stack trace:
    ...
    : While sending batch, nums: 62, files: /work6/clickhouse/data/default/dist/shard1_replica1/66827258.bin

As you can see "Sending a batch of 10 files" but "nums: 62"

Fixes: #23856
Refs: #41813
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
 2023-02-03 13:54:40 +01:00
Alexey Milovidov
3e3df376c0
Merge pull request #45995 from CurtizJ/check-dynamic-columns 
Check dynamic columns of part before its commit
 2023-02-03 15:39:54 +03:00
Sema Checherinda
e0db071563
Merge branch 'master' into release-merge-blocker-drop-part 2023-02-03 13:35:51 +01:00
Sema Checherinda
fe0b13e9f4
Merge pull request #45458 from FrankChen021/better_postpone_reason 
Show the name of replica that is executing a merge in the postpone_reason
 2023-02-03 13:15:10 +01:00
Kseniia Sumarokova
ef7acb9b66
Merge pull request #44404 from kssenii/storage-rabbitmq-fix-flush 
RabbitMQ - fix writing many small blocks by flushing data only exactly by flush_interval_ms or by max_block_size
 2023-02-03 11:50:12 +01:00
Smita Kulkarni
123f3cfe1f Used unordered_set for log entries and fixed the callback function - Updated checking of SYSTEM SYNC REPLICA 2023-02-03 08:48:16 +01:00
Frank Chen
d38adfab30 Merge two overridden functions as one 2023-02-03 15:27:45 +08:00
flynn
d119367d7d fix 2023-02-03 07:06:58 +00:00
Alexey Milovidov
108d1c3442
Merge pull request #45872 from ClickHouse/fix-trash-query-kind 
Remove trash, add a feature and fix a bug
 2023-02-03 09:58:43 +03:00
chen
07a6de1713
Merge branch 'master' into s3_file_not_found 2023-02-03 12:29:17 +08:00
xiedeyantu
562642ab7f add settings s3_allow_throw_if_mismatch_files 2023-02-03 12:27:13 +08:00
Frank Chen
d3a05a11da Merge remote-tracking branch 'remotes/github/master' into stack_trace_in_part_log 2023-02-03 11:39:43 +08:00
Frank Chen
7af5ae7f2e
Merge branch 'master' into better_postpone_reason 2023-02-03 11:20:37 +08:00
Anton Popov
8ca602a148 check dynamic columns of part before its commit 2023-02-03 02:58:28 +00:00
kssenii
a391de18a1 Merge remote-tracking branch 'upstream/master' into allow-single-disk-instead-of-storage-policy 2023-02-02 20:49:41 +01:00
flynn
2d1dd694c6 make _table LowCardinality 2023-02-02 16:33:31 +00:00
flynn
f88a8bac19 fix 2023-02-02 16:22:09 +00:00
flynn
bc38ebaf52 use LowCardnality for _part and _partition_id virtual column 
fix
 2023-02-02 16:20:29 +00:00
Alexander Gololobov
ec3bb0c04e
Merge pull request #45794 from ClickHouse/parallelize_is_storage_touched_by_mutations 
Do not restrict count() query to 1 thread in isStorageTouchedByMutations()
 2023-02-02 16:44:11 +01:00
Kseniia Sumarokova
6705fafef9
Merge branch 'master' into storage-rabbitmq-fix-flush 2023-02-02 16:06:51 +01:00
Sema Checherinda
44be0f41dd
Merge branch 'master' into release-merge-blocker-drop-part 2023-02-02 14:12:40 +01:00
xiedeyantu
e22cc0eb98 Throw an error on no files satisfying S3 wildcard 2023-02-02 19:13:34 +08:00
Alexey Milovidov
62bfdb349a Merge branch 'master' into fix-trash-query-kind 2023-02-02 04:26:49 +01:00
Sema Checherinda
7c416477ac do not run wal on remote disks 2023-02-01 20:03:45 +01:00
kssenii
bbf060bf89 Fix test 2023-02-01 18:22:05 +01:00
Sema Checherinda
3e1b17e4cf
fix build 2023-02-01 17:09:43 +01:00
Anton Popov
9f26e16963 loose the assertion 2023-02-01 15:31:06 +00:00
Anton Popov
8dd6dbc452
Merge pull request #45857 from CurtizJ/fix-minor-mistake 
Fix minor mistake after refactoring
 2023-02-01 15:33:19 +01:00
Sema Checherinda
41a1dddc7a
Merge branch 'master' into better_postpone_reason 2023-02-01 15:16:34 +01:00
Alexander Tokmakov
aba4a00a17 fix 2023-02-01 14:47:57 +01:00
Sema Checherinda
db2aa48c91 do not block merges when old parts are droping in drop queries 2023-02-01 13:30:56 +01:00
Vladimir C
0d48eed68f
Merge pull request #45848 from Algunenano/fix_connection_timeouts_context_include 2023-02-01 12:53:50 +01:00
Robert Schulze
b512316586
Merge pull request #45682 from ClickHouse/rename-qrc-to-qc 
Rename "Query Result Cache" to "Query Cache"
 2023-02-01 11:23:29 +01:00
Alexander Gololobov
e705f11ad1
Merge pull request #45786 from ClickHouse/delete_on_cluster 
Support DELETE ON CLUSTER
 2023-02-01 10:52:01 +01:00
Alexey Milovidov
9b908a0be0 Fix trash around query kind, part 2 2023-02-01 03:31:31 +01:00
Alexey Milovidov
5c4db119fa Fix trash around query kind, part 1 2023-02-01 03:11:54 +01:00
Vitaly Baranov
96b140cc95 Rename columns in system.backups: num_files, num_processed_files, processed_files_size 
num_processed_files -> num_files (BACKUP) / files_read (RESTORE)
processed_files_size -> total_size (BACKUP) / bytes_read (RESTORE)
 2023-01-31 22:45:41 +01:00